Jul 14 21:51:03.621032 ip-10-0-141-219 systemd[1]: Starting Kubernetes Kubelet... Jul 14 21:51:06.591622 ip-10-0-141-219 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 21:51:06.591622 ip-10-0-141-219 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 21:51:06.591622 ip-10-0-141-219 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 21:51:06.591622 ip-10-0-141-219 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 21:51:06.591622 ip-10-0-141-219 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 21:51:06.592295 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.591675 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 21:51:06.596192 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596164 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:51:06.596192 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596186 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:51:06.596192 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596190 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:51:06.596192 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596194 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:51:06.596192 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596199 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596205 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596208 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596211 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596214 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596217 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596220 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596223 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596226 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596228 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596231 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596233 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596236 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596238 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596241 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596244 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596253 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596256 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596259 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596261 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:51:06.596394 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596264 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596266 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596269 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596272 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596274 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596277 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596280 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596282 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596285 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596287 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596290 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596294 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596297 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596300 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596305 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596308 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596311 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596314 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596316 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596319 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:51:06.596858 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596322 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596326 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596330 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596334 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596337 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596341 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596344 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596347 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596350 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596352 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596355 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596358 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596360 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596363 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596365 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596368 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596370 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596373 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596375 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:51:06.597355 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596378 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596381 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596383 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596386 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596390 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596392 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596395 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596398 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596401 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596403 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596406 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596409 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596412 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596414 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596416 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596419 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596429 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596432 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596435 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596438 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:51:06.597787 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596441 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596444 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596446 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596449 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596451 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596454 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596457 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596460 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596462 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596465 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596467 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596470 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596473 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596475 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596478 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.596480 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597209 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597218 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597221 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597224 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:51:06.598259 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597227 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597230 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597233 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597236 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597238 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597240 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597244 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597246 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597249 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597251 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597254 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597257 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597259 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597263 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597265 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597268 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597270 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597273 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597277 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:51:06.598717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597279 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597282 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597284 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597287 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597289 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597293 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597296 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597298 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597301 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597304 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597306 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597310 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597312 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597315 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597318 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597320 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597323 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597325 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597328 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:51:06.599179 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597332 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597334 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597337 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597339 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597342 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597345 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597347 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597350 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597353 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597356 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597359 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597362 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597365 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597368 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597371 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597373 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597376 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597379 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597381 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597384 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:51:06.599622 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597387 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597389 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597392 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597394 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597398 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597401 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597404 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597406 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597409 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597411 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597413 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597416 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597418 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597423 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597427 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597431 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597435 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597438 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597441 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:51:06.600109 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597444 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597450 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597453 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597456 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597458 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597461 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597464 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597468 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597470 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597473 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597476 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597478 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597481 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597484 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597486 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597489 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597492 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597494 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597887 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597893 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:51:06.600564 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597896 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597898 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597901 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597903 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597906 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597909 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597911 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597914 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597917 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597920 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597922 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597925 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597928 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597934 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597938 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597941 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597944 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597947 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597949 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597952 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:51:06.601067 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597955 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597957 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597960 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597963 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597965 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597968 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597970 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597972 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597975 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597978 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597981 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597984 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597987 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597990 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597992 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597995 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.597997 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598000 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598002 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:51:06.601569 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598005 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598009 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598012 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598015 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598018 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598036 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598042 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598045 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598048 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598051 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598053 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598056 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598058 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598061 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598063 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598068 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598071 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598074 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598076 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:51:06.602038 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598079 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598081 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598084 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598086 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598089 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598091 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598094 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598096 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598099 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598102 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598104 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598106 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598109 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598111 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598114 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598116 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598119 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598121 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598123 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598127 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:51:06.602493 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598130 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598132 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598136 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598138 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598141 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598143 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598146 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598148 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598154 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598157 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598160 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598163 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598166 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598168 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598171 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598174 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598177 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598179 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598182 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598253 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 21:51:06.603009 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598261 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598267 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598271 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598276 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598279 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598283 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598292 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598295 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598297 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598301 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598304 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598309 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598312 2562 flags.go:64] FLAG: --cgroup-root="" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598315 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598318 2562 flags.go:64] FLAG: --client-ca-file="" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598321 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598324 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598328 2562 flags.go:64] FLAG: --cluster-domain="" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598331 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598334 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598338 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598342 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598345 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 21:51:06.603500 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598348 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598351 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598354 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598357 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598360 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598363 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598366 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598369 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598373 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598376 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598379 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598382 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598385 2562 flags.go:64] FLAG: --enable-server="true" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598387 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598392 2562 flags.go:64] FLAG: --event-burst="100" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598395 2562 flags.go:64] FLAG: --event-qps="50" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598398 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598401 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598404 2562 flags.go:64] FLAG: --eviction-hard="" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598408 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598412 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598415 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598418 2562 flags.go:64] FLAG: --eviction-soft="" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598421 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598423 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 21:51:06.604065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598427 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598430 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598433 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598436 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598441 2562 flags.go:64] FLAG: --feature-gates="" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598444 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598447 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598450 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598453 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598457 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598460 2562 flags.go:64] FLAG: --help="false" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598463 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-141-219.ec2.internal" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598466 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598468 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598471 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598475 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598478 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598481 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598484 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598487 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598490 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598493 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598496 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598499 2562 flags.go:64] FLAG: --kube-reserved="" Jul 14 21:51:06.604667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598502 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598504 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598507 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598511 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598514 2562 flags.go:64] FLAG: --lock-file="" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598517 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598520 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598523 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598528 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598532 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598534 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598537 2562 flags.go:64] FLAG: --logging-format="text" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598541 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598545 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598547 2562 flags.go:64] FLAG: --manifest-url="" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598550 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598554 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598557 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598562 2562 flags.go:64] FLAG: --max-pods="110" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598566 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598568 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598571 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598574 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598577 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598580 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 21:51:06.605286 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598583 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598591 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598594 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598597 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598599 2562 flags.go:64] FLAG: --pod-cidr="" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598602 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598607 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598610 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598613 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598616 2562 flags.go:64] FLAG: --port="10250" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598619 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598623 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a6c2a328431a1dca" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598627 2562 flags.go:64] FLAG: --qos-reserved="" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598630 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598633 2562 flags.go:64] FLAG: --register-node="true" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598635 2562 flags.go:64] FLAG: --register-with-taints="" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598639 2562 flags.go:64] FLAG: --registry-burst="10" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598642 2562 flags.go:64] FLAG: --registry-qps="5" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598645 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598650 2562 flags.go:64] FLAG: --reserved-memory="" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598653 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598656 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598660 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598662 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598665 2562 flags.go:64] FLAG: --runonce="false" Jul 14 21:51:06.605877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598668 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598671 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598674 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598677 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598680 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598683 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598686 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598689 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598691 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598694 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598697 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598700 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598703 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598706 2562 flags.go:64] FLAG: --system-cgroups="" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598708 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598713 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598716 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598719 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598747 2562 flags.go:64] FLAG: --tls-min-version="" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598750 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598753 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598756 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598759 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598762 2562 flags.go:64] FLAG: --v="2" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598766 2562 flags.go:64] FLAG: --version="false" Jul 14 21:51:06.606544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598771 2562 flags.go:64] FLAG: --vmodule="" Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598776 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.598780 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598891 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598895 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598898 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598901 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598903 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598906 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598908 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598911 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598913 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598915 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598918 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598921 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598923 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598926 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598928 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598931 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598933 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:51:06.607160 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598936 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598940 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598943 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598946 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598950 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598952 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598956 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598960 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598962 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598965 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598968 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598970 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598973 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598978 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598981 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598984 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598986 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598988 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598991 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:51:06.607685 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598994 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598996 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.598999 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599001 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599004 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599006 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599009 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599011 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599014 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599016 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599032 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599036 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599038 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599041 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599043 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599046 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599048 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599051 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599053 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599056 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:51:06.608155 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599058 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599061 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599063 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599065 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599068 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599071 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599075 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599078 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599081 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599083 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599086 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599088 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599091 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599093 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599096 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599099 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599101 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599103 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599106 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599109 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:51:06.608651 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599111 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599114 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599116 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599119 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599122 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599124 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599126 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599130 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599134 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599137 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599140 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599143 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599145 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599148 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599151 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599154 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599157 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599160 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599163 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599166 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:51:06.609208 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599169 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599172 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.599175 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.599961 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.606880 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.606900 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.606922 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.606931 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.606991 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.606998 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607001 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607005 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607008 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607011 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:51:06.609782 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607014 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607016 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607035 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607039 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607041 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607044 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607047 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607050 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607053 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607055 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607058 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607061 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607063 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607066 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607068 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607071 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607073 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607076 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607079 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607081 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:51:06.610304 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607084 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607087 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607089 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607099 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607102 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607105 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607108 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607110 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607113 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607116 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607118 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607121 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607124 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607126 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607131 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607134 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607137 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607140 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607143 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:51:06.610810 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607145 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607148 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607150 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607153 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607156 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607158 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607161 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607163 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607166 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607169 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607172 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607174 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607177 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607181 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607185 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607187 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607191 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607200 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607203 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:51:06.611281 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607206 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607209 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607211 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607214 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607217 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607220 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607228 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607231 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607234 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607237 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607239 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607242 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607244 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607247 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607250 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607252 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607255 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607257 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607260 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607262 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:51:06.611805 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607265 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607267 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607270 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607272 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607275 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607277 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607280 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607282 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607284 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607287 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607290 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607299 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607302 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607304 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607307 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:51:06.612363 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.607311 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607446 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607450 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607454 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607456 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607459 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607462 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607465 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607468 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607470 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607473 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607476 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607478 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607481 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607485 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607488 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607491 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607494 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607496 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:51:06.612740 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607500 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607503 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607505 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607508 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607510 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607513 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607515 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607518 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607521 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607524 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607532 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607535 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607537 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607540 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607543 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607545 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607548 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607551 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607553 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607556 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:51:06.613255 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607558 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607561 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607563 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607566 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607569 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607571 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607573 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607576 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607578 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607581 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607583 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607586 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607588 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607590 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607593 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607596 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607598 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607600 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607603 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607607 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:51:06.613994 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607610 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607613 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607615 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607626 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607628 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607631 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607634 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607637 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607639 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607642 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607644 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607646 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607649 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607651 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607654 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607656 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607659 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607661 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607664 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607666 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:51:06.614875 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607669 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607671 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607674 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607676 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607678 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607681 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607683 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607686 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607689 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607692 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607695 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607698 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607701 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607703 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607707 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607711 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607719 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607722 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607725 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607728 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:51:06.615537 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:06.607730 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:51:06.616287 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.607735 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 21:51:06.616287 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.607858 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 21:51:06.616287 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.609957 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 21:51:06.616287 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.611442 2562 server.go:1019] "Starting client certificate rotation" Jul 14 21:51:06.616287 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.611559 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 21:51:06.616287 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.611606 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 21:51:06.616287 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.614201 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 21:51:06.619270 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.619250 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 21:51:06.627819 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.627800 2562 log.go:25] "Validated CRI v1 runtime API" Jul 14 21:51:06.629990 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.629973 2562 log.go:25] "Validated CRI v1 image API" Jul 14 21:51:06.631398 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.631379 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 21:51:06.635501 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.635479 2562 fs.go:135] Filesystem UUIDs: map[136a41fb-0e52-4169-a965-322863bf1e8e:/dev/nvme0n1p4 5deb215e-f6a4-476c-9939-000eb60318d6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 14 21:51:06.635572 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.635500 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 21:51:06.641079 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.640945 2562 manager.go:216] Machine: {Timestamp:2026-07-14 21:51:06.639605283 +0000 UTC m=+1.489075313 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119959 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f8ae108c343c2dd30ff2a5d0a163b SystemUUID:ec2f8ae1-08c3-43c2-dd30-ff2a5d0a163b BootID:bfee5f12-59f3-4706-871a-afce83c1e045 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b0:d8:bc:b2:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b0:d8:bc:b2:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:64:14:63:f2:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 21:51:06.641079 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.641073 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 21:51:06.641194 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.641163 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 21:51:06.643896 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.643867 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 21:51:06.644056 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.643900 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 21:51:06.644992 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.644981 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 21:51:06.645048 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.644994 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 21:51:06.645048 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.645001 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 21:51:06.646244 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.646233 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 21:51:06.647422 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.647412 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 21:51:06.649797 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.649784 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 14 21:51:06.649934 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.649925 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 21:51:06.649970 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.649955 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 14 21:51:06.649970 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.649965 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 21:51:06.650036 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.649979 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 21:51:06.650036 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.650014 2562 kubelet.go:398] "Adding apiserver pod source" Jul 14 21:51:06.650087 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.650039 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 21:51:06.650621 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.650607 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 21:51:06.650679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.650646 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 21:51:06.651495 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.651467 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 21:51:06.654628 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.654604 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 21:51:06.656381 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.656365 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 21:51:06.656458 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.656396 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 21:51:06.656613 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.656600 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 21:51:06.656647 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.656622 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 21:51:06.656647 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.656632 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 21:51:06.656647 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.656640 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 21:51:06.656775 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.656653 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 21:51:06.656775 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.656662 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 21:51:06.657115 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.657103 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 21:51:06.657115 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.657115 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 21:51:06.657213 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.657122 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 21:51:06.657213 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.657128 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 21:51:06.657213 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.657137 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 21:51:06.657213 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.657145 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 21:51:06.657757 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.657745 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 21:51:06.657757 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.657756 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 21:51:06.659774 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.659749 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 21:51:06.659862 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.659844 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 21:51:06.660257 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.660244 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-219.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 21:51:06.661247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.661232 2562 server.go:1268] "Started kubelet" Jul 14 21:51:06.661722 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.661308 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 21:51:06.661722 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.661316 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 21:51:06.661722 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.661390 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 21:51:06.662235 ip-10-0-141-219 systemd[1]: Started Kubernetes Kubelet. Jul 14 21:51:06.662850 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.662535 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 14 21:51:06.662850 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.662669 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 21:51:06.667289 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.667263 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 21:51:06.667289 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.667271 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 21:51:06.667405 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.667372 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:06.667605 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.667585 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 21:51:06.667853 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.667835 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 21:51:06.667922 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.667856 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 21:51:06.667922 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.667909 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 21:51:06.668016 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.668003 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 14 21:51:06.668016 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.668011 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 14 21:51:06.668377 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.668360 2562 factory.go:55] Registering systemd factory Jul 14 21:51:06.668455 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.668384 2562 factory.go:223] Registration of the systemd container factory successfully Jul 14 21:51:06.668671 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.668653 2562 factory.go:153] Registering CRI-O factory Jul 14 21:51:06.668755 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.668677 2562 factory.go:223] Registration of the crio container factory successfully Jul 14 21:51:06.668755 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.668741 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 21:51:06.668847 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.668767 2562 factory.go:103] Registering Raw factory Jul 14 21:51:06.668847 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.668786 2562 manager.go:1196] Started watching for new ooms in manager Jul 14 21:51:06.669288 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.669273 2562 manager.go:349] Starting recovery of all containers Jul 14 21:51:06.671527 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.671500 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 21:51:06.671630 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.671611 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 21:51:06.672608 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.671416 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-219.ec2.internal.18c246740e3d8c9e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-219.ec2.internal,UID:ip-10-0-141-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-219.ec2.internal,},FirstTimestamp:2026-07-14 21:51:06.661203102 +0000 UTC m=+1.510673133,LastTimestamp:2026-07-14 21:51:06.661203102 +0000 UTC m=+1.510673133,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-219.ec2.internal,}" Jul 14 21:51:06.684249 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.684230 2562 manager.go:354] Recovery completed Jul 14 21:51:06.688911 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.688897 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:51:06.690928 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.690908 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:51:06.691000 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.690940 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:51:06.691000 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.690950 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:51:06.691499 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.691481 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 21:51:06.691558 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.691499 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 21:51:06.691558 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.691518 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 14 21:51:06.691616 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.691482 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:51:06.692263 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.692249 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:51:06.692315 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.692278 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:51:06.692315 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.692290 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:51:06.692675 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.692662 2562 policy_none.go:49] "None policy: Start" Jul 14 21:51:06.692710 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.692683 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 21:51:06.692710 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.692699 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 21:51:06.692769 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.692710 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ntgs2" Jul 14 21:51:06.693597 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.693585 2562 policy_none.go:47] "Start" Jul 14 21:51:06.693636 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.693550 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-219.ec2.internal.18c24674100317f2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-219.ec2.internal,UID:ip-10-0-141-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-219.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-219.ec2.internal,},FirstTimestamp:2026-07-14 21:51:06.690926578 +0000 UTC m=+1.540396608,LastTimestamp:2026-07-14 21:51:06.690926578 +0000 UTC m=+1.540396608,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-219.ec2.internal,}" Jul 14 21:51:06.704134 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.703819 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-219.ec2.internal.18c2467410035ddd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-219.ec2.internal,UID:ip-10-0-141-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-219.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-219.ec2.internal,},FirstTimestamp:2026-07-14 21:51:06.690944477 +0000 UTC m=+1.540414508,LastTimestamp:2026-07-14 21:51:06.690944477 +0000 UTC m=+1.540414508,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-219.ec2.internal,}" Jul 14 21:51:06.706263 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.706244 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ntgs2" Jul 14 21:51:06.731758 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.731738 2562 manager.go:337] "Starting Device Plugin manager" Jul 14 21:51:06.731877 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.731784 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 21:51:06.731877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.731793 2562 server.go:87] "Starting device plugin registration server" Jul 14 21:51:06.732136 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.732123 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 21:51:06.732172 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.732142 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 21:51:06.732240 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.732214 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 21:51:06.732387 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.732308 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 21:51:06.732387 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.732321 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 21:51:06.733119 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.733093 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 21:51:06.733226 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.733161 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:06.802160 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.802126 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 21:51:06.803524 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.803368 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 21:51:06.803524 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.803402 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 21:51:06.803524 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.803433 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 21:51:06.803524 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.803466 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 21:51:06.806218 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.806195 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 21:51:06.833321 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.833292 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:51:06.834624 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.834606 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:51:06.834732 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.834638 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:51:06.834732 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.834650 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:51:06.834732 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.834678 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-219.ec2.internal" Jul 14 21:51:06.845356 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.845333 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-219.ec2.internal" Jul 14 21:51:06.845441 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.845360 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-219.ec2.internal\": node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:06.884618 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.884589 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:06.903875 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.903844 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal"] Jul 14 21:51:06.903990 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.903923 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:51:06.904935 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.904918 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:51:06.904994 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.904952 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:51:06.904994 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.904968 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:51:06.906654 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.906639 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:51:06.906794 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.906780 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" Jul 14 21:51:06.906855 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.906818 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:51:06.907647 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.907628 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:51:06.907759 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.907664 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:51:06.907759 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.907678 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:51:06.907759 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.907631 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:51:06.907759 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.907736 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:51:06.907759 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.907747 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:51:06.909362 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.909343 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal" Jul 14 21:51:06.909463 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.909376 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:51:06.910173 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.910159 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:51:06.910251 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.910183 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:51:06.910251 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.910197 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:51:06.930987 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.930962 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-219.ec2.internal\" not found" node="ip-10-0-141-219.ec2.internal" Jul 14 21:51:06.935735 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.935715 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-219.ec2.internal\" not found" node="ip-10-0-141-219.ec2.internal" Jul 14 21:51:06.969835 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.969742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72e6bc3cf174f91a838415868d2b5b35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal\" (UID: \"72e6bc3cf174f91a838415868d2b5b35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" Jul 14 21:51:06.969835 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.969782 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72e6bc3cf174f91a838415868d2b5b35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal\" (UID: \"72e6bc3cf174f91a838415868d2b5b35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" Jul 14 21:51:06.969835 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:06.969820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/355c9c11aac4261fc69af718f808c587-config\") pod \"kube-apiserver-proxy-ip-10-0-141-219.ec2.internal\" (UID: \"355c9c11aac4261fc69af718f808c587\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal" Jul 14 21:51:06.984969 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:06.984941 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:07.070397 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.070366 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72e6bc3cf174f91a838415868d2b5b35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal\" (UID: \"72e6bc3cf174f91a838415868d2b5b35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" Jul 14 21:51:07.070397 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.070402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72e6bc3cf174f91a838415868d2b5b35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal\" (UID: \"72e6bc3cf174f91a838415868d2b5b35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" Jul 14 21:51:07.070616 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.070424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/355c9c11aac4261fc69af718f808c587-config\") pod \"kube-apiserver-proxy-ip-10-0-141-219.ec2.internal\" (UID: \"355c9c11aac4261fc69af718f808c587\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal" Jul 14 21:51:07.070616 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.070481 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/355c9c11aac4261fc69af718f808c587-config\") pod \"kube-apiserver-proxy-ip-10-0-141-219.ec2.internal\" (UID: \"355c9c11aac4261fc69af718f808c587\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal" Jul 14 21:51:07.070616 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.070496 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72e6bc3cf174f91a838415868d2b5b35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal\" (UID: \"72e6bc3cf174f91a838415868d2b5b35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" Jul 14 21:51:07.070616 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.070482 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72e6bc3cf174f91a838415868d2b5b35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal\" (UID: \"72e6bc3cf174f91a838415868d2b5b35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" Jul 14 21:51:07.086010 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:07.085966 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:07.186921 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:07.186889 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:07.233134 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.233073 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" Jul 14 21:51:07.238441 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.238425 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal" Jul 14 21:51:07.287905 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:07.287874 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:07.388339 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:07.388305 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:07.488871 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:07.488796 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:07.589582 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:07.589543 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:07.610724 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.610695 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 21:51:07.611213 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.610896 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 21:51:07.690112 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:07.690075 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:07.709269 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.709219 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 21:46:06 +0000 UTC" deadline="2028-03-25 09:12:23.13044596 +0000 UTC" Jul 14 21:51:07.709269 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.709267 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14867h21m15.421182572s" Jul 14 21:51:07.755008 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:07.754959 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355c9c11aac4261fc69af718f808c587.slice/crio-0a1c60749e96d70a1c23c7bbffc111d11223197d817d4a574745fe5200f24c9a WatchSource:0}: Error finding container 0a1c60749e96d70a1c23c7bbffc111d11223197d817d4a574745fe5200f24c9a: Status 404 returned error can't find the container with id 0a1c60749e96d70a1c23c7bbffc111d11223197d817d4a574745fe5200f24c9a Jul 14 21:51:07.758759 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.758745 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 21:51:07.790637 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:07.790594 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:07.805904 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.805872 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:51:07.807091 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.807070 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:51:07.807171 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.807103 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:51:07.807171 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.807113 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:51:07.808779 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.808521 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal" event={"ID":"355c9c11aac4261fc69af718f808c587","Type":"ContainerStarted","Data":"0a1c60749e96d70a1c23c7bbffc111d11223197d817d4a574745fe5200f24c9a"} Jul 14 21:51:07.822587 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.822559 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 21:51:07.881986 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:07.881960 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 21:51:07.891328 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:07.891303 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:07.991843 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:07.991792 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-219.ec2.internal\" not found" Jul 14 21:51:08.024665 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.024640 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 21:51:08.068827 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.068793 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" Jul 14 21:51:08.088960 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.088929 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 21:51:08.090261 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.090247 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal" Jul 14 21:51:08.106871 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.106842 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 21:51:08.170646 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:08.170621 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e6bc3cf174f91a838415868d2b5b35.slice/crio-67980b37444fef2e4415465fe7926190dd280c404004ec26bfb57fc803a06989 WatchSource:0}: Error finding container 67980b37444fef2e4415465fe7926190dd280c404004ec26bfb57fc803a06989: Status 404 returned error can't find the container with id 67980b37444fef2e4415465fe7926190dd280c404004ec26bfb57fc803a06989 Jul 14 21:51:08.651859 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.651822 2562 apiserver.go:52] "Watching apiserver" Jul 14 21:51:08.660100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.660072 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 21:51:08.660525 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.660498 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-8dxrp","openshift-dns/node-resolver-b4ttn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal","openshift-multus/multus-additional-cni-plugins-cf2lf","openshift-network-operator/iptables-alerter-dpsz2","openshift-ovn-kubernetes/ovnkube-node-47d2f","kube-system/konnectivity-agent-rw49t","kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb","openshift-cluster-node-tuning-operator/tuned-h7r2w","openshift-image-registry/node-ca-frswj","openshift-multus/multus-hsjvk","openshift-multus/network-metrics-daemon-xpch8","openshift-network-diagnostics/network-check-target-6flwq"] Jul 14 21:51:08.662932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.662913 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.665300 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.665272 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.665993 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.665967 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:51:08.666107 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.666043 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 21:51:08.666107 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.666055 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-58c7n\"" Jul 14 21:51:08.667401 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.667383 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:08.667503 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.667462 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:08.667727 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.667708 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 21:51:08.668010 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.667991 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 21:51:08.668297 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.668252 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 21:51:08.668383 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.668326 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n79fv\"" Jul 14 21:51:08.668557 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.668538 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 21:51:08.669686 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.669668 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.671954 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.671935 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:08.673188 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.673167 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 21:51:08.673715 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.673697 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 21:51:08.673808 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.673788 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mwdtm\"" Jul 14 21:51:08.674279 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.674260 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 21:51:08.674378 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.674265 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 21:51:08.674585 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.674569 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.675829 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.675811 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 21:51:08.675924 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.675819 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 21:51:08.676650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.676635 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:51:08.676790 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.676772 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.676980 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.676968 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 21:51:08.677218 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.677200 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n6wxj\"" Jul 14 21:51:08.680499 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.679902 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.680598 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48818c69-846c-4964-946c-5dff3e23d337-tmp\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.680598 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/535d8ec6-5b58-40eb-b09f-b8b3a8044071-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.680598 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680593 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.680750 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-modprobe-d\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.680750 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680655 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-kubernetes\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.680750 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-var-lib-kubelet\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.680750 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680725 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-run\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.680937 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680753 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-host\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.680937 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xtq\" (UniqueName: \"kubernetes.io/projected/48818c69-846c-4964-946c-5dff3e23d337-kube-api-access-s8xtq\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.680937 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680873 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38b6f451-563f-498b-86d9-18dbd2f78850-serviceca\") pod \"node-ca-frswj\" (UID: \"38b6f451-563f-498b-86d9-18dbd2f78850\") " pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.681111 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/535d8ec6-5b58-40eb-b09f-b8b3a8044071-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.681111 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.680982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjmtr\" (UniqueName: \"kubernetes.io/projected/535d8ec6-5b58-40eb-b09f-b8b3a8044071-kube-api-access-pjmtr\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.681111 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-sysctl-d\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.681111 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-cnibin\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.681111 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/535d8ec6-5b58-40eb-b09f-b8b3a8044071-cni-binary-copy\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.681338 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681210 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 21:51:08.681338 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681218 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 21:51:08.681338 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681228 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48818c69-846c-4964-946c-5dff3e23d337-etc-tuned\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.681338 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681258 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-system-cni-dir\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.681338 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681286 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-os-release\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.681338 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681310 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-systemd\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.681338 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38b6f451-563f-498b-86d9-18dbd2f78850-host\") pod \"node-ca-frswj\" (UID: \"38b6f451-563f-498b-86d9-18dbd2f78850\") " pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.681655 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681364 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-sysconfig\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.681655 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681395 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbtd\" (UniqueName: \"kubernetes.io/projected/38b6f451-563f-498b-86d9-18dbd2f78850-kube-api-access-xzbtd\") pod \"node-ca-frswj\" (UID: \"38b6f451-563f-498b-86d9-18dbd2f78850\") " pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.681655 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kcfj\" (UniqueName: \"kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj\") pod \"network-check-target-6flwq\" (UID: \"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70\") " pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:08.681655 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-sysctl-conf\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.681655 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-sys\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.681655 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681582 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-lib-modules\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.681915 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681751 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 21:51:08.681915 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681909 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 21:51:08.682013 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.681932 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 21:51:08.682357 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.682339 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 21:51:08.682692 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.682672 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fstfb\"" Jul 14 21:51:08.682910 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.682892 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 21:51:08.683182 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.683137 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kq9lm\"" Jul 14 21:51:08.683278 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.683229 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 21:51:08.683278 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.683259 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:08.683390 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.683276 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 21:51:08.684139 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.684110 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6bdc7\"" Jul 14 21:51:08.685560 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.685455 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 21:51:08.686661 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.686643 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:08.686769 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.686708 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:08.689128 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.689108 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:08.689214 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.689149 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gdxrb\"" Jul 14 21:51:08.689388 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.689367 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 21:51:08.689703 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.689663 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 21:51:08.691343 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.691242 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:08.691343 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.691308 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:08.693433 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.693414 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zk6j6\"" Jul 14 21:51:08.693521 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.693422 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 21:51:08.694257 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.694239 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 21:51:08.707397 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.707371 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 21:51:08.723892 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.723865 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6jmfh" Jul 14 21:51:08.733452 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.733428 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6jmfh" Jul 14 21:51:08.768790 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.768763 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 21:51:08.781822 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.781789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16b760c4-0184-4c7b-a1cc-65e0d7841740-env-overrides\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.781992 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.781867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48818c69-846c-4964-946c-5dff3e23d337-etc-tuned\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.781992 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.781899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-system-cni-dir\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.781992 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.781920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-os-release\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.781992 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.781939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-run-ovn\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.781992 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.781961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-cni-bin\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.781992 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.781983 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfhg5\" (UniqueName: \"kubernetes.io/projected/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-kube-api-access-vfhg5\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-system-cni-dir\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782007 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-system-cni-dir\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-systemd\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782061 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-os-release\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782085 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38b6f451-563f-498b-86d9-18dbd2f78850-host\") pod \"node-ca-frswj\" (UID: \"38b6f451-563f-498b-86d9-18dbd2f78850\") " pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782127 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-systemd\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782154 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38b6f451-563f-498b-86d9-18dbd2f78850-host\") pod \"node-ca-frswj\" (UID: \"38b6f451-563f-498b-86d9-18dbd2f78850\") " pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782170 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-log-socket\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782178 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782246 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbtd\" (UniqueName: \"kubernetes.io/projected/38b6f451-563f-498b-86d9-18dbd2f78850-kube-api-access-xzbtd\") pod \"node-ca-frswj\" (UID: \"38b6f451-563f-498b-86d9-18dbd2f78850\") " pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782270 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-systemd-units\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.782324 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-socket-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782410 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-registration-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782436 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-run-netns\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782475 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-sysctl-conf\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782512 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-sys\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-lib-modules\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/535d8ec6-5b58-40eb-b09f-b8b3a8044071-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782583 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-sys\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782587 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-kubelet\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782624 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-hostroot\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782668 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3383a4e-85df-440d-9ab7-ba09e045389a-konnectivity-ca\") pod \"konnectivity-agent-rw49t\" (UID: \"a3383a4e-85df-440d-9ab7-ba09e045389a\") " pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782678 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-sysctl-conf\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-kubernetes\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-run-ovn-kubernetes\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-cni-netd\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.782963 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782774 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxlv\" (UniqueName: \"kubernetes.io/projected/9e23d05f-8c0f-40cb-91bc-44790bd8965f-kube-api-access-dhxlv\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-kubernetes\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-lib-modules\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6kk5\" (UniqueName: \"kubernetes.io/projected/ea9ea5a7-f98e-455b-bd1a-c57400d5bad2-kube-api-access-r6kk5\") pod \"iptables-alerter-dpsz2\" (UID: \"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2\") " pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-run\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-host\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782931 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-run-systemd\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-node-log\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782998 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-run\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.782999 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-host\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5svx\" (UniqueName: \"kubernetes.io/projected/16b760c4-0184-4c7b-a1cc-65e0d7841740-kube-api-access-p5svx\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783083 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-device-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7f4d121-c802-43d7-995b-4a54e3d71205-dbus\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-cnibin\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38b6f451-563f-498b-86d9-18dbd2f78850-serviceca\") pod \"node-ca-frswj\" (UID: \"38b6f451-563f-498b-86d9-18dbd2f78850\") " pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/535d8ec6-5b58-40eb-b09f-b8b3a8044071-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.783887 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-slash\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-run-netns\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783239 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-daemon-config\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783272 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/535d8ec6-5b58-40eb-b09f-b8b3a8044071-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783294 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6212dead-9e15-43dc-bbef-57928fbd39f1-tmp-dir\") pod \"node-resolver-b4ttn\" (UID: \"6212dead-9e15-43dc-bbef-57928fbd39f1\") " pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgcb\" (UniqueName: \"kubernetes.io/projected/6212dead-9e15-43dc-bbef-57928fbd39f1-kube-api-access-kfgcb\") pod \"node-resolver-b4ttn\" (UID: \"6212dead-9e15-43dc-bbef-57928fbd39f1\") " pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-sysctl-d\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-cnibin\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783387 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-etc-openvswitch\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16b760c4-0184-4c7b-a1cc-65e0d7841740-ovnkube-config\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783416 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-sysctl-d\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16b760c4-0184-4c7b-a1cc-65e0d7841740-ovn-node-metrics-cert\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/535d8ec6-5b58-40eb-b09f-b8b3a8044071-cnibin\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-run-multus-certs\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783500 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.784682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783549 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/535d8ec6-5b58-40eb-b09f-b8b3a8044071-cni-binary-copy\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-var-lib-openvswitch\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783598 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-var-lib-kubelet\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783617 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-cni-dir\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783622 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38b6f451-563f-498b-86d9-18dbd2f78850-serviceca\") pod \"node-ca-frswj\" (UID: \"38b6f451-563f-498b-86d9-18dbd2f78850\") " pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-cni-binary-copy\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783673 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-sysconfig\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783687 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/535d8ec6-5b58-40eb-b09f-b8b3a8044071-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcfj\" (UniqueName: \"kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj\") pod \"network-check-target-6flwq\" (UID: \"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70\") " pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783725 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783744 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-os-release\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783760 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48818c69-846c-4964-946c-5dff3e23d337-tmp\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783732 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-sysconfig\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16b760c4-0184-4c7b-a1cc-65e0d7841740-ovnkube-script-lib\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783829 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea9ea5a7-f98e-455b-bd1a-c57400d5bad2-iptables-alerter-script\") pod \"iptables-alerter-dpsz2\" (UID: \"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2\") " pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-modprobe-d\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783884 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-var-lib-kubelet\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.785479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-sys-fs\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783916 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/535d8ec6-5b58-40eb-b09f-b8b3a8044071-cni-binary-copy\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783933 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea9ea5a7-f98e-455b-bd1a-c57400d5bad2-host-slash\") pod \"iptables-alerter-dpsz2\" (UID: \"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2\") " pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-run-k8s-cni-cncf-io\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783998 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-etc-modprobe-d\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.783999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-var-lib-cni-bin\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784045 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48818c69-846c-4964-946c-5dff3e23d337-var-lib-kubelet\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784053 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-var-lib-cni-multus\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xtq\" (UniqueName: \"kubernetes.io/projected/48818c69-846c-4964-946c-5dff3e23d337-kube-api-access-s8xtq\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784112 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7f4d121-c802-43d7-995b-4a54e3d71205-kubelet-config\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784137 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-conf-dir\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784160 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-etc-kubernetes\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784186 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6212dead-9e15-43dc-bbef-57928fbd39f1-hosts-file\") pod \"node-resolver-b4ttn\" (UID: \"6212dead-9e15-43dc-bbef-57928fbd39f1\") " pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784210 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3383a4e-85df-440d-9ab7-ba09e045389a-agent-certs\") pod \"konnectivity-agent-rw49t\" (UID: \"a3383a4e-85df-440d-9ab7-ba09e045389a\") " pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjmtr\" (UniqueName: \"kubernetes.io/projected/535d8ec6-5b58-40eb-b09f-b8b3a8044071-kube-api-access-pjmtr\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-run-openvswitch\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784290 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-socket-dir-parent\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.786247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.784313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfv4\" (UniqueName: \"kubernetes.io/projected/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-kube-api-access-xmfv4\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.786906 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.785986 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48818c69-846c-4964-946c-5dff3e23d337-etc-tuned\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.786906 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.786078 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48818c69-846c-4964-946c-5dff3e23d337-tmp\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.793203 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.793176 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:51:08.793306 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.793206 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:51:08.793306 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.793222 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4kcfj for pod openshift-network-diagnostics/network-check-target-6flwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:08.793306 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.793298 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj podName:448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:09.29326866 +0000 UTC m=+4.142738688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcfj" (UniqueName: "kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj") pod "network-check-target-6flwq" (UID: "448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:08.793850 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.793830 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbtd\" (UniqueName: \"kubernetes.io/projected/38b6f451-563f-498b-86d9-18dbd2f78850-kube-api-access-xzbtd\") pod \"node-ca-frswj\" (UID: \"38b6f451-563f-498b-86d9-18dbd2f78850\") " pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.798369 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.798306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xtq\" (UniqueName: \"kubernetes.io/projected/48818c69-846c-4964-946c-5dff3e23d337-kube-api-access-s8xtq\") pod \"tuned-h7r2w\" (UID: \"48818c69-846c-4964-946c-5dff3e23d337\") " pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.799059 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.798643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjmtr\" (UniqueName: \"kubernetes.io/projected/535d8ec6-5b58-40eb-b09f-b8b3a8044071-kube-api-access-pjmtr\") pod \"multus-additional-cni-plugins-cf2lf\" (UID: \"535d8ec6-5b58-40eb-b09f-b8b3a8044071\") " pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:08.810883 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.810848 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" event={"ID":"72e6bc3cf174f91a838415868d2b5b35","Type":"ContainerStarted","Data":"67980b37444fef2e4415465fe7926190dd280c404004ec26bfb57fc803a06989"} Jul 14 21:51:08.884780 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.884729 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-run-systemd\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.884780 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.884775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-node-log\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.885017 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.884831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-node-log\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.885017 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.884849 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-run-systemd\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.885017 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.884900 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5svx\" (UniqueName: \"kubernetes.io/projected/16b760c4-0184-4c7b-a1cc-65e0d7841740-kube-api-access-p5svx\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.885017 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.884951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-device-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.885017 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.884977 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7f4d121-c802-43d7-995b-4a54e3d71205-dbus\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:08.885017 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-cnibin\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-slash\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-device-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885101 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-slash\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885123 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-cnibin\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-run-netns\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-daemon-config\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885199 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7f4d121-c802-43d7-995b-4a54e3d71205-dbus\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885244 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-run-netns\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885259 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6212dead-9e15-43dc-bbef-57928fbd39f1-tmp-dir\") pod \"node-resolver-b4ttn\" (UID: \"6212dead-9e15-43dc-bbef-57928fbd39f1\") " pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgcb\" (UniqueName: \"kubernetes.io/projected/6212dead-9e15-43dc-bbef-57928fbd39f1-kube-api-access-kfgcb\") pod \"node-resolver-b4ttn\" (UID: \"6212dead-9e15-43dc-bbef-57928fbd39f1\") " pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-etc-openvswitch\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885377 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16b760c4-0184-4c7b-a1cc-65e0d7841740-ovnkube-config\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885408 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16b760c4-0184-4c7b-a1cc-65e0d7841740-ovn-node-metrics-cert\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885431 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-run-multus-certs\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885455 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.886100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-var-lib-openvswitch\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885537 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-var-lib-openvswitch\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885549 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6212dead-9e15-43dc-bbef-57928fbd39f1-tmp-dir\") pod \"node-resolver-b4ttn\" (UID: \"6212dead-9e15-43dc-bbef-57928fbd39f1\") " pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885563 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-run-multus-certs\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885599 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885602 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-etc-openvswitch\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885709 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-var-lib-kubelet\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-cni-dir\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-daemon-config\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-var-lib-kubelet\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-cni-binary-copy\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-cni-dir\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885888 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-os-release\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885941 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16b760c4-0184-4c7b-a1cc-65e0d7841740-ovnkube-script-lib\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea9ea5a7-f98e-455b-bd1a-c57400d5bad2-iptables-alerter-script\") pod \"iptables-alerter-dpsz2\" (UID: \"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2\") " pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.885994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-sys-fs\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.886679 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.886003 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea9ea5a7-f98e-455b-bd1a-c57400d5bad2-host-slash\") pod \"iptables-alerter-dpsz2\" (UID: \"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2\") " pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886063 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-run-k8s-cni-cncf-io\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-var-lib-cni-bin\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.886101 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs podName:885409c4-463f-4bc0-a3dc-b9f5a25d37e4 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:09.386082819 +0000 UTC m=+4.235552855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs") pod "network-metrics-daemon-xpch8" (UID: "885409c4-463f-4bc0-a3dc-b9f5a25d37e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-var-lib-cni-multus\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886128 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-var-lib-cni-bin\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7f4d121-c802-43d7-995b-4a54e3d71205-kubelet-config\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886194 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-conf-dir\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16b760c4-0184-4c7b-a1cc-65e0d7841740-ovnkube-config\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886206 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-os-release\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886221 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-etc-kubernetes\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886246 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6212dead-9e15-43dc-bbef-57928fbd39f1-hosts-file\") pod \"node-resolver-b4ttn\" (UID: \"6212dead-9e15-43dc-bbef-57928fbd39f1\") " pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886252 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7f4d121-c802-43d7-995b-4a54e3d71205-kubelet-config\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886271 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3383a4e-85df-440d-9ab7-ba09e045389a-agent-certs\") pod \"konnectivity-agent-rw49t\" (UID: \"a3383a4e-85df-440d-9ab7-ba09e045389a\") " pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886286 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-var-lib-cni-multus\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-run-openvswitch\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.887448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886300 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-cni-binary-copy\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886323 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-socket-dir-parent\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886340 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-sys-fs\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfv4\" (UniqueName: \"kubernetes.io/projected/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-kube-api-access-xmfv4\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886360 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6212dead-9e15-43dc-bbef-57928fbd39f1-hosts-file\") pod \"node-resolver-b4ttn\" (UID: \"6212dead-9e15-43dc-bbef-57928fbd39f1\") " pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16b760c4-0184-4c7b-a1cc-65e0d7841740-env-overrides\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886413 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-etc-kubernetes\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-run-ovn\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-cni-bin\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886448 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-run-openvswitch\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhg5\" (UniqueName: \"kubernetes.io/projected/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-kube-api-access-vfhg5\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-system-cni-dir\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886598 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-run-ovn\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886640 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-cni-bin\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-log-socket\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16b760c4-0184-4c7b-a1cc-65e0d7841740-ovnkube-script-lib\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16b760c4-0184-4c7b-a1cc-65e0d7841740-env-overrides\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea9ea5a7-f98e-455b-bd1a-c57400d5bad2-host-slash\") pod \"iptables-alerter-dpsz2\" (UID: \"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2\") " pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:08.888231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886904 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-run-k8s-cni-cncf-io\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886904 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea9ea5a7-f98e-455b-bd1a-c57400d5bad2-iptables-alerter-script\") pod \"iptables-alerter-dpsz2\" (UID: \"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2\") " pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886982 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-systemd-units\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887008 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-socket-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.886983 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-log-socket\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-system-cni-dir\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887047 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-systemd-units\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-registration-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-run-netns\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887087 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-conf-dir\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887112 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-kubelet\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887111 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-registration-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887132 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887148 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887136 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-multus-socket-dir-parent\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887166 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-host-run-netns\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.889083 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-hostroot\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3383a4e-85df-440d-9ab7-ba09e045389a-konnectivity-ca\") pod \"konnectivity-agent-rw49t\" (UID: \"a3383a4e-85df-440d-9ab7-ba09e045389a\") " pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.887209 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-run-ovn-kubernetes\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887259 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-cni-netd\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:08.887272 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret podName:c7f4d121-c802-43d7-995b-4a54e3d71205 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:09.387255021 +0000 UTC m=+4.236725052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret") pod "global-pull-secret-syncer-8dxrp" (UID: "c7f4d121-c802-43d7-995b-4a54e3d71205") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887169 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-kubelet\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887292 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-cni-netd\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887210 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e23d05f-8c0f-40cb-91bc-44790bd8965f-socket-dir\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887302 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxlv\" (UniqueName: \"kubernetes.io/projected/9e23d05f-8c0f-40cb-91bc-44790bd8965f-kube-api-access-dhxlv\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887205 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-hostroot\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6kk5\" (UniqueName: \"kubernetes.io/projected/ea9ea5a7-f98e-455b-bd1a-c57400d5bad2-kube-api-access-r6kk5\") pod \"iptables-alerter-dpsz2\" (UID: \"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2\") " pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887328 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16b760c4-0184-4c7b-a1cc-65e0d7841740-host-run-ovn-kubernetes\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.887660 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3383a4e-85df-440d-9ab7-ba09e045389a-konnectivity-ca\") pod \"konnectivity-agent-rw49t\" (UID: \"a3383a4e-85df-440d-9ab7-ba09e045389a\") " pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.889621 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3383a4e-85df-440d-9ab7-ba09e045389a-agent-certs\") pod \"konnectivity-agent-rw49t\" (UID: \"a3383a4e-85df-440d-9ab7-ba09e045389a\") " pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:08.889824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.889767 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16b760c4-0184-4c7b-a1cc-65e0d7841740-ovn-node-metrics-cert\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.909077 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.908991 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5svx\" (UniqueName: \"kubernetes.io/projected/16b760c4-0184-4c7b-a1cc-65e0d7841740-kube-api-access-p5svx\") pod \"ovnkube-node-47d2f\" (UID: \"16b760c4-0184-4c7b-a1cc-65e0d7841740\") " pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:08.909077 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.908993 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgcb\" (UniqueName: \"kubernetes.io/projected/6212dead-9e15-43dc-bbef-57928fbd39f1-kube-api-access-kfgcb\") pod \"node-resolver-b4ttn\" (UID: \"6212dead-9e15-43dc-bbef-57928fbd39f1\") " pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:08.909261 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.909129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6kk5\" (UniqueName: \"kubernetes.io/projected/ea9ea5a7-f98e-455b-bd1a-c57400d5bad2-kube-api-access-r6kk5\") pod \"iptables-alerter-dpsz2\" (UID: \"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2\") " pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:08.909617 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.909590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhg5\" (UniqueName: \"kubernetes.io/projected/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-kube-api-access-vfhg5\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:08.910534 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.910503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxlv\" (UniqueName: \"kubernetes.io/projected/9e23d05f-8c0f-40cb-91bc-44790bd8965f-kube-api-access-dhxlv\") pod \"aws-ebs-csi-driver-node-gtsxb\" (UID: \"9e23d05f-8c0f-40cb-91bc-44790bd8965f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:08.911356 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.911336 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfv4\" (UniqueName: \"kubernetes.io/projected/106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1-kube-api-access-xmfv4\") pod \"multus-hsjvk\" (UID: \"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1\") " pod="openshift-multus/multus-hsjvk" Jul 14 21:51:08.951280 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.951241 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 21:51:08.974263 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.974233 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" Jul 14 21:51:08.983879 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.983832 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-frswj" Jul 14 21:51:08.993523 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:08.993489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" Jul 14 21:51:09.001014 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.000993 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dpsz2" Jul 14 21:51:09.008981 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.008930 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:09.016303 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.016282 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" Jul 14 21:51:09.025065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.025045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hsjvk" Jul 14 21:51:09.032373 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.032349 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b4ttn" Jul 14 21:51:09.039494 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.039436 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:09.390520 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.390480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:09.390714 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.390553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcfj\" (UniqueName: \"kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj\") pod \"network-check-target-6flwq\" (UID: \"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70\") " pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:09.390714 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.390582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:09.390714 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:09.390610 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:09.390714 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:09.390684 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret podName:c7f4d121-c802-43d7-995b-4a54e3d71205 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:10.390663823 +0000 UTC m=+5.240133856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret") pod "global-pull-secret-syncer-8dxrp" (UID: "c7f4d121-c802-43d7-995b-4a54e3d71205") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:09.390714 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:09.390691 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:09.390956 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:09.390738 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:51:09.390956 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:09.390766 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs podName:885409c4-463f-4bc0-a3dc-b9f5a25d37e4 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:10.390750161 +0000 UTC m=+5.240220184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs") pod "network-metrics-daemon-xpch8" (UID: "885409c4-463f-4bc0-a3dc-b9f5a25d37e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:09.390956 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:09.390766 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:51:09.390956 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:09.390789 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4kcfj for pod openshift-network-diagnostics/network-check-target-6flwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:09.390956 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:09.390825 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj podName:448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:10.390814104 +0000 UTC m=+5.240284126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcfj" (UniqueName: "kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj") pod "network-check-target-6flwq" (UID: "448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:09.737664 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.737572 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 21:46:08 +0000 UTC" deadline="2027-12-16 08:43:08.308942966 +0000 UTC" Jul 14 21:51:09.737664 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.737609 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12466h51m58.57133891s" Jul 14 21:51:09.804161 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.804123 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:09.804332 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:09.804133 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:09.804398 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:09.804345 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:09.804398 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:09.804225 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:09.834106 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:09.834074 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106a12ec_b8ba_4b6e_bcc7_37b0c8bd89c1.slice/crio-6d7579e2cc5df5a779b476a205769201f960fb3b83d30aeba86d53035e3d758b WatchSource:0}: Error finding container 6d7579e2cc5df5a779b476a205769201f960fb3b83d30aeba86d53035e3d758b: Status 404 returned error can't find the container with id 6d7579e2cc5df5a779b476a205769201f960fb3b83d30aeba86d53035e3d758b Jul 14 21:51:09.835565 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:09.835528 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e23d05f_8c0f_40cb_91bc_44790bd8965f.slice/crio-a1039b1bffaf70234f21aa49d1db086a777e171918f87579fd962dede6b8145f WatchSource:0}: Error finding container a1039b1bffaf70234f21aa49d1db086a777e171918f87579fd962dede6b8145f: Status 404 returned error can't find the container with id a1039b1bffaf70234f21aa49d1db086a777e171918f87579fd962dede6b8145f Jul 14 21:51:09.835821 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:09.835804 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b760c4_0184_4c7b_a1cc_65e0d7841740.slice/crio-859718e21174ed9294f7f88f9a81e31b476573f1490c823befba092674282f07 WatchSource:0}: Error finding container 859718e21174ed9294f7f88f9a81e31b476573f1490c823befba092674282f07: Status 404 returned error can't find the container with id 859718e21174ed9294f7f88f9a81e31b476573f1490c823befba092674282f07 Jul 14 21:51:09.837731 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:09.837500 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b6f451_563f_498b_86d9_18dbd2f78850.slice/crio-463c136784de110eeb9e92c2d03642bf6717a3cc5d2842ad4ed9638edb466d49 WatchSource:0}: Error finding container 463c136784de110eeb9e92c2d03642bf6717a3cc5d2842ad4ed9638edb466d49: Status 404 returned error can't find the container with id 463c136784de110eeb9e92c2d03642bf6717a3cc5d2842ad4ed9638edb466d49 Jul 14 21:51:09.840532 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:09.840500 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9ea5a7_f98e_455b_bd1a_c57400d5bad2.slice/crio-e7ff9491fd62223069f1c938f1bcbe25bb7c69df27745a59c1bc81a17bf70cb2 WatchSource:0}: Error finding container e7ff9491fd62223069f1c938f1bcbe25bb7c69df27745a59c1bc81a17bf70cb2: Status 404 returned error can't find the container with id e7ff9491fd62223069f1c938f1bcbe25bb7c69df27745a59c1bc81a17bf70cb2 Jul 14 21:51:09.841570 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:09.841542 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535d8ec6_5b58_40eb_b09f_b8b3a8044071.slice/crio-5fb16efc625f3e9038ca6c9778987080ce3dd1183c523307e46e1294091ba2d9 WatchSource:0}: Error finding container 5fb16efc625f3e9038ca6c9778987080ce3dd1183c523307e46e1294091ba2d9: Status 404 returned error can't find the container with id 5fb16efc625f3e9038ca6c9778987080ce3dd1183c523307e46e1294091ba2d9 Jul 14 21:51:09.842429 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:09.842378 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3383a4e_85df_440d_9ab7_ba09e045389a.slice/crio-33484d13b60d401c247714f24121afa5917ed9d3961a9752db6da0b9343e162c WatchSource:0}: Error finding container 33484d13b60d401c247714f24121afa5917ed9d3961a9752db6da0b9343e162c: Status 404 returned error can't find the container with id 33484d13b60d401c247714f24121afa5917ed9d3961a9752db6da0b9343e162c Jul 14 21:51:09.843748 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:09.843521 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48818c69_846c_4964_946c_5dff3e23d337.slice/crio-9320c4bcc35c1430dc875ce06d347da7cb516dbc71ad14ec2150edf8425ae5c7 WatchSource:0}: Error finding container 9320c4bcc35c1430dc875ce06d347da7cb516dbc71ad14ec2150edf8425ae5c7: Status 404 returned error can't find the container with id 9320c4bcc35c1430dc875ce06d347da7cb516dbc71ad14ec2150edf8425ae5c7 Jul 14 21:51:09.844945 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:09.844891 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6212dead_9e15_43dc_bbef_57928fbd39f1.slice/crio-ea01c4d651fe509c160a9377c7519d2b5d529d0e8ead475a9c126cbc444d8bfc WatchSource:0}: Error finding container ea01c4d651fe509c160a9377c7519d2b5d529d0e8ead475a9c126cbc444d8bfc: Status 404 returned error can't find the container with id ea01c4d651fe509c160a9377c7519d2b5d529d0e8ead475a9c126cbc444d8bfc Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.401116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.401460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcfj\" (UniqueName: \"kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj\") pod \"network-check-target-6flwq\" (UID: \"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70\") " pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.401493 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:10.401633 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:10.401701 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs podName:885409c4-463f-4bc0-a3dc-b9f5a25d37e4 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:12.401679691 +0000 UTC m=+7.251149729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs") pod "network-metrics-daemon-xpch8" (UID: "885409c4-463f-4bc0-a3dc-b9f5a25d37e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:10.402129 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:10.402181 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret podName:c7f4d121-c802-43d7-995b-4a54e3d71205 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:12.402166142 +0000 UTC m=+7.251636174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret") pod "global-pull-secret-syncer-8dxrp" (UID: "c7f4d121-c802-43d7-995b-4a54e3d71205") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:10.402265 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:10.402279 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:10.402293 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4kcfj for pod openshift-network-diagnostics/network-check-target-6flwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:10.402389 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:10.402349 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj podName:448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:12.402338172 +0000 UTC m=+7.251808194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcfj" (UniqueName: "kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj") pod "network-check-target-6flwq" (UID: "448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:10.738412 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.738300 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 21:46:08 +0000 UTC" deadline="2028-02-17 08:31:08.474130581 +0000 UTC" Jul 14 21:51:10.738412 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.738363 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13978h39m57.735771366s" Jul 14 21:51:10.807753 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.807188 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:10.807753 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:10.807330 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:10.840683 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.840609 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" event={"ID":"535d8ec6-5b58-40eb-b09f-b8b3a8044071","Type":"ContainerStarted","Data":"5fb16efc625f3e9038ca6c9778987080ce3dd1183c523307e46e1294091ba2d9"} Jul 14 21:51:10.847752 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.847719 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" event={"ID":"9e23d05f-8c0f-40cb-91bc-44790bd8965f","Type":"ContainerStarted","Data":"a1039b1bffaf70234f21aa49d1db086a777e171918f87579fd962dede6b8145f"} Jul 14 21:51:10.864354 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.864321 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hsjvk" event={"ID":"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1","Type":"ContainerStarted","Data":"6d7579e2cc5df5a779b476a205769201f960fb3b83d30aeba86d53035e3d758b"} Jul 14 21:51:10.888707 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.888672 2562 generic.go:356] "Generic (PLEG): container finished" podID="72e6bc3cf174f91a838415868d2b5b35" containerID="e1f6bff62533d17ef8884a6a3b9fdddfd84711c92deb45b83ab2a96325d7a873" exitCode=0 Jul 14 21:51:10.888861 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.888763 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" event={"ID":"72e6bc3cf174f91a838415868d2b5b35","Type":"ContainerDied","Data":"e1f6bff62533d17ef8884a6a3b9fdddfd84711c92deb45b83ab2a96325d7a873"} Jul 14 21:51:10.892410 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.892367 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rw49t" event={"ID":"a3383a4e-85df-440d-9ab7-ba09e045389a","Type":"ContainerStarted","Data":"33484d13b60d401c247714f24121afa5917ed9d3961a9752db6da0b9343e162c"} Jul 14 21:51:10.901806 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.901772 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dpsz2" event={"ID":"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2","Type":"ContainerStarted","Data":"e7ff9491fd62223069f1c938f1bcbe25bb7c69df27745a59c1bc81a17bf70cb2"} Jul 14 21:51:10.918382 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.918226 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-frswj" event={"ID":"38b6f451-563f-498b-86d9-18dbd2f78850","Type":"ContainerStarted","Data":"463c136784de110eeb9e92c2d03642bf6717a3cc5d2842ad4ed9638edb466d49"} Jul 14 21:51:10.946167 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.946011 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" event={"ID":"16b760c4-0184-4c7b-a1cc-65e0d7841740","Type":"ContainerStarted","Data":"859718e21174ed9294f7f88f9a81e31b476573f1490c823befba092674282f07"} Jul 14 21:51:10.975125 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.975089 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal" event={"ID":"355c9c11aac4261fc69af718f808c587","Type":"ContainerStarted","Data":"5416f0d4512d936066b678631b7787fc07d7cfae63144bf4dcbaf49b21f367cb"} Jul 14 21:51:10.996652 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:10.996568 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b4ttn" event={"ID":"6212dead-9e15-43dc-bbef-57928fbd39f1","Type":"ContainerStarted","Data":"ea01c4d651fe509c160a9377c7519d2b5d529d0e8ead475a9c126cbc444d8bfc"} Jul 14 21:51:11.004227 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:11.004192 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" event={"ID":"48818c69-846c-4964-946c-5dff3e23d337","Type":"ContainerStarted","Data":"9320c4bcc35c1430dc875ce06d347da7cb516dbc71ad14ec2150edf8425ae5c7"} Jul 14 21:51:11.803939 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:11.803909 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:11.804441 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:11.804066 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:11.804713 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:11.804691 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:11.804817 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:11.804796 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:12.030574 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:12.030493 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" event={"ID":"72e6bc3cf174f91a838415868d2b5b35","Type":"ContainerStarted","Data":"a09f0d784c905254ffab6e26656133664b3d7ad9cc95bd03bcde1a81425043da"} Jul 14 21:51:12.048724 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:12.048675 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-219.ec2.internal" podStartSLOduration=4.048656261 podStartE2EDuration="4.048656261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:11.004063826 +0000 UTC m=+5.853533867" watchObservedRunningTime="2026-07-14 21:51:12.048656261 +0000 UTC m=+6.898126598" Jul 14 21:51:12.420244 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:12.420203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcfj\" (UniqueName: \"kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj\") pod \"network-check-target-6flwq\" (UID: \"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70\") " pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:12.420445 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:12.420258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:12.420445 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:12.420319 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:12.420549 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:12.420506 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:12.420597 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:12.420578 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret podName:c7f4d121-c802-43d7-995b-4a54e3d71205 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:16.420553192 +0000 UTC m=+11.270023216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret") pod "global-pull-secret-syncer-8dxrp" (UID: "c7f4d121-c802-43d7-995b-4a54e3d71205") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:12.421074 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:12.421052 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:51:12.421074 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:12.421078 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:51:12.421232 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:12.421091 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4kcfj for pod openshift-network-diagnostics/network-check-target-6flwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:12.421232 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:12.421138 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj podName:448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:16.421123273 +0000 UTC m=+11.270593304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcfj" (UniqueName: "kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj") pod "network-check-target-6flwq" (UID: "448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:12.421232 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:12.421217 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:12.421394 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:12.421269 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs podName:885409c4-463f-4bc0-a3dc-b9f5a25d37e4 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:16.421239676 +0000 UTC m=+11.270709693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs") pod "network-metrics-daemon-xpch8" (UID: "885409c4-463f-4bc0-a3dc-b9f5a25d37e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:12.806891 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:12.806854 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:12.807349 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:12.807047 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:13.804587 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:13.804553 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:13.804784 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:13.804553 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:13.804784 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:13.804711 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:13.804784 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:13.804733 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:14.804881 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:14.804336 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:14.804881 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:14.804507 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:15.804426 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:15.804388 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:15.804426 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:15.804420 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:15.804646 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:15.804547 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:15.804713 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:15.804657 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:16.457085 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:16.456257 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:16.457085 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:16.456333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcfj\" (UniqueName: \"kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj\") pod \"network-check-target-6flwq\" (UID: \"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70\") " pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:16.457085 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:16.456362 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:16.457085 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:16.456503 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:16.457085 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:16.456564 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs podName:885409c4-463f-4bc0-a3dc-b9f5a25d37e4 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:24.456546545 +0000 UTC m=+19.306016568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs") pod "network-metrics-daemon-xpch8" (UID: "885409c4-463f-4bc0-a3dc-b9f5a25d37e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:16.457085 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:16.456939 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:16.457085 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:16.456988 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret podName:c7f4d121-c802-43d7-995b-4a54e3d71205 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:24.456974262 +0000 UTC m=+19.306444300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret") pod "global-pull-secret-syncer-8dxrp" (UID: "c7f4d121-c802-43d7-995b-4a54e3d71205") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:16.457085 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:16.457091 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:51:16.457835 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:16.457108 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:51:16.457835 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:16.457119 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4kcfj for pod openshift-network-diagnostics/network-check-target-6flwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:16.457835 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:16.457153 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj podName:448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:24.457141331 +0000 UTC m=+19.306611367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcfj" (UniqueName: "kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj") pod "network-check-target-6flwq" (UID: "448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:16.808643 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:16.806004 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:16.808643 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:16.806173 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:17.804674 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:17.804639 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:17.805148 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:17.804693 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:17.805148 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:17.804785 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:17.805148 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:17.804913 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:18.804659 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:18.804620 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:18.804836 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:18.804771 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:19.804430 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:19.804396 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:19.804615 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:19.804397 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:19.804615 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:19.804506 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:19.804615 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:19.804580 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:20.804704 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:20.804668 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:20.805172 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:20.804800 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:21.803835 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:21.803803 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:21.803835 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:21.803846 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:21.804125 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:21.803933 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:21.804125 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:21.804083 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:22.804674 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:22.804640 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:22.805162 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:22.804775 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:23.804509 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:23.804474 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:23.804509 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:23.804497 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:23.805074 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:23.804591 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:23.805074 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:23.804705 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:24.522113 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:24.522043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:24.522285 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:24.522147 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcfj\" (UniqueName: \"kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj\") pod \"network-check-target-6flwq\" (UID: \"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70\") " pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:24.522285 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:24.522167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:24.522285 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:24.522207 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:24.522285 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:24.522274 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:24.522285 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:24.522289 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret podName:c7f4d121-c802-43d7-995b-4a54e3d71205 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:40.522269256 +0000 UTC m=+35.371739276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret") pod "global-pull-secret-syncer-8dxrp" (UID: "c7f4d121-c802-43d7-995b-4a54e3d71205") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:24.522548 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:24.522324 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs podName:885409c4-463f-4bc0-a3dc-b9f5a25d37e4 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:40.522306767 +0000 UTC m=+35.371776790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs") pod "network-metrics-daemon-xpch8" (UID: "885409c4-463f-4bc0-a3dc-b9f5a25d37e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:24.522548 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:24.522324 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:51:24.522548 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:24.522348 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:51:24.522548 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:24.522362 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4kcfj for pod openshift-network-diagnostics/network-check-target-6flwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:24.522548 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:24.522413 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj podName:448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70 nodeName:}" failed. No retries permitted until 2026-07-14 21:51:40.522398153 +0000 UTC m=+35.371868171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcfj" (UniqueName: "kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj") pod "network-check-target-6flwq" (UID: "448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:24.804695 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:24.804617 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:24.804854 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:24.804755 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:25.804203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:25.804164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:25.804394 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:25.804293 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:25.804394 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:25.804344 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:25.804479 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:25.804452 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:26.804902 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:26.804868 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:26.805471 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:26.804981 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:27.803897 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:27.803853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:27.804094 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:27.803904 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:27.804094 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:27.803998 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:27.804212 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:27.804139 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:28.804923 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:28.804491 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:28.805530 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:28.805052 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:29.070385 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.070351 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" event={"ID":"16b760c4-0184-4c7b-a1cc-65e0d7841740","Type":"ContainerStarted","Data":"d5d080822db9bcc4c07514c0d6f83d2667600aeb22080b094856876ba32da4f7"} Jul 14 21:51:29.070552 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.070435 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" event={"ID":"16b760c4-0184-4c7b-a1cc-65e0d7841740","Type":"ContainerStarted","Data":"3ab1c61e7a3cf7a8bfb72c392fe753be4775e5924ce6f7a0a0ca83761b85d7dd"} Jul 14 21:51:29.070552 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.070485 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" event={"ID":"16b760c4-0184-4c7b-a1cc-65e0d7841740","Type":"ContainerStarted","Data":"3bcb73dff68ec10ab7eb4c6d28135a0096eb730ad4837216bfc4976130297e86"} Jul 14 21:51:29.070552 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.070532 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" event={"ID":"16b760c4-0184-4c7b-a1cc-65e0d7841740","Type":"ContainerStarted","Data":"e5ae914c0adc18cd9ec1f7aa778984a4463b15edad2eb761dcfc29dd2be5a8e0"} Jul 14 21:51:29.070711 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.070578 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" event={"ID":"16b760c4-0184-4c7b-a1cc-65e0d7841740","Type":"ContainerStarted","Data":"6ac7da3a3c48dedfcadcb40ed93ba69848b2f837e78ef22bec2957f19df9ad9a"} Jul 14 21:51:29.070711 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.070625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" event={"ID":"16b760c4-0184-4c7b-a1cc-65e0d7841740","Type":"ContainerStarted","Data":"7aaba9503b580f381bc465671bb6782b2f36d96891b7a9f560f4192608b2ac48"} Jul 14 21:51:29.071951 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.071923 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b4ttn" event={"ID":"6212dead-9e15-43dc-bbef-57928fbd39f1","Type":"ContainerStarted","Data":"b54a549c27419a31b2f8e9a7720c5c1ed0dcf0841f0237a497b8620b497267da"} Jul 14 21:51:29.073530 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.073498 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" event={"ID":"48818c69-846c-4964-946c-5dff3e23d337","Type":"ContainerStarted","Data":"d5c003d4893201827ec122b18872fc93f8ea8052fc3cf68dc5d1c955ee35058e"} Jul 14 21:51:29.075291 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.075262 2562 generic.go:356] "Generic (PLEG): container finished" podID="535d8ec6-5b58-40eb-b09f-b8b3a8044071" containerID="718be1515224b1fb536f67c36aa9e0b73e595964b8820ade078bd53b8975b7bb" exitCode=0 Jul 14 21:51:29.075394 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.075341 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" event={"ID":"535d8ec6-5b58-40eb-b09f-b8b3a8044071","Type":"ContainerDied","Data":"718be1515224b1fb536f67c36aa9e0b73e595964b8820ade078bd53b8975b7bb"} Jul 14 21:51:29.076961 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.076939 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" event={"ID":"9e23d05f-8c0f-40cb-91bc-44790bd8965f","Type":"ContainerStarted","Data":"d50625e647b3acdc8fa70c686cd96dc7d6b88daa19cf92dfa27fdcf6670fef39"} Jul 14 21:51:29.078557 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.078534 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hsjvk" event={"ID":"106a12ec-b8ba-4b6e-bcc7-37b0c8bd89c1","Type":"ContainerStarted","Data":"9219054c03c9df89fceb56aa96c8ff61375d5d3d204d08f550fd352deb8cd1d6"} Jul 14 21:51:29.080272 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.080248 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rw49t" event={"ID":"a3383a4e-85df-440d-9ab7-ba09e045389a","Type":"ContainerStarted","Data":"a274dc64b8acb51a6e604822453bfa4176dc12209bd2928f688684fcadec82e7"} Jul 14 21:51:29.081898 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.081876 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-frswj" event={"ID":"38b6f451-563f-498b-86d9-18dbd2f78850","Type":"ContainerStarted","Data":"69e693a39bcc909216b4eb113e89a6568a82a3224a8698376ed53a3533c93f76"} Jul 14 21:51:29.090731 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.090687 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b4ttn" podStartSLOduration=4.701125629 podStartE2EDuration="23.090675861s" totalImagesPullingTime="18.389550232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:29.090498841 +0000 UTC m=+23.939968879" watchObservedRunningTime="2026-07-14 21:51:29.090675861 +0000 UTC m=+23.940145899" Jul 14 21:51:29.090834 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.090768 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-219.ec2.internal" podStartSLOduration=21.090762877 podStartE2EDuration="21.090762877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:12.049563711 +0000 UTC m=+6.899033767" watchObservedRunningTime="2026-07-14 21:51:29.090762877 +0000 UTC m=+23.940232916" Jul 14 21:51:29.106785 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.106743 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h7r2w" podStartSLOduration=4.715182043 podStartE2EDuration="23.106729772s" totalImagesPullingTime="18.391547729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:29.106318129 +0000 UTC m=+23.955788179" watchObservedRunningTime="2026-07-14 21:51:29.106729772 +0000 UTC m=+23.956200062" Jul 14 21:51:29.120287 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.120244 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-frswj" podStartSLOduration=4.722823068 podStartE2EDuration="23.120230671s" totalImagesPullingTime="18.397407603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:29.119589202 +0000 UTC m=+23.969059235" watchObservedRunningTime="2026-07-14 21:51:29.120230671 +0000 UTC m=+23.969700709" Jul 14 21:51:29.136553 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.136506 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hsjvk" podStartSLOduration=4.67969889 podStartE2EDuration="23.136493484s" totalImagesPullingTime="18.456794594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:29.136439066 +0000 UTC m=+23.985909104" watchObservedRunningTime="2026-07-14 21:51:29.136493484 +0000 UTC m=+23.985963523" Jul 14 21:51:29.174621 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.174573 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rw49t" podStartSLOduration=11.815396334 podStartE2EDuration="22.17455554s" totalImagesPullingTime="10.359159206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:29.174118677 +0000 UTC m=+24.023588716" watchObservedRunningTime="2026-07-14 21:51:29.17455554 +0000 UTC m=+24.024025578" Jul 14 21:51:29.793157 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.793111 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 21:51:29.804548 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.804527 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:29.804640 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:29.804611 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:29.804681 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:29.804661 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:29.804730 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:29.804717 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:30.086451 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:30.086412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" event={"ID":"9e23d05f-8c0f-40cb-91bc-44790bd8965f","Type":"ContainerStarted","Data":"6e9f8775c310cc18b912ba7674346c8f14367f33b1763c2e43c63aef50444b4f"} Jul 14 21:51:30.088195 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:30.088167 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dpsz2" event={"ID":"ea9ea5a7-f98e-455b-bd1a-c57400d5bad2","Type":"ContainerStarted","Data":"9b4d0967718c9d59ddfbf909ce77e4ff3bc2bd497e8e41a7c7f749da71bef210"} Jul 14 21:51:30.104480 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:30.104424 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dpsz2" podStartSLOduration=5.709992779 podStartE2EDuration="24.104406096s" totalImagesPullingTime="18.394413317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:30.10411109 +0000 UTC m=+24.953581130" watchObservedRunningTime="2026-07-14 21:51:30.104406096 +0000 UTC m=+24.953876134" Jul 14 21:51:30.747442 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:30.746643 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T21:51:29.793134164Z","UUID":"f0445126-9329-4d2e-9398-37f8bfcf2a95","Handler":null,"Name":"","Endpoint":""} Jul 14 21:51:30.748582 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:30.748559 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 21:51:30.748752 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:30.748605 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 21:51:30.803968 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:30.803939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:30.804173 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:30.804093 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:31.093196 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:31.093158 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" event={"ID":"9e23d05f-8c0f-40cb-91bc-44790bd8965f","Type":"ContainerStarted","Data":"173201c870bc96cddcb736e8f2f700f6510206d4ffed95dbfa8dc6ce0b0f865d"} Jul 14 21:51:31.109947 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:31.109871 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtsxb" podStartSLOduration=4.557899047 podStartE2EDuration="25.109855318s" totalImagesPullingTime="20.551956271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:31.109294434 +0000 UTC m=+25.958764475" watchObservedRunningTime="2026-07-14 21:51:31.109855318 +0000 UTC m=+25.959325356" Jul 14 21:51:31.804056 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:31.804010 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:31.804056 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:31.804057 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:31.804285 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:31.804139 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:31.804285 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:31.804258 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:32.099544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:32.099452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" event={"ID":"16b760c4-0184-4c7b-a1cc-65e0d7841740","Type":"ContainerStarted","Data":"f8d4f25f483d5a711c99d30d882abb6cb337eb6f355503e61a24d15c5cda1bc0"} Jul 14 21:51:32.804297 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:32.804102 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:32.804462 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:32.804416 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:33.305987 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:33.305955 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:33.306543 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:33.306523 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:33.804235 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:33.804059 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:33.804416 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:33.804085 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:33.804416 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:33.804326 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:33.804416 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:33.804405 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:34.106351 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:34.106266 2562 generic.go:356] "Generic (PLEG): container finished" podID="535d8ec6-5b58-40eb-b09f-b8b3a8044071" containerID="89e4d63c2fc0188345799dec6b96d32490e11ddb4b5bb8603c791862ed5a922b" exitCode=0 Jul 14 21:51:34.106351 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:34.106341 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" event={"ID":"535d8ec6-5b58-40eb-b09f-b8b3a8044071","Type":"ContainerDied","Data":"89e4d63c2fc0188345799dec6b96d32490e11ddb4b5bb8603c791862ed5a922b"} Jul 14 21:51:34.109933 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:34.109912 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" event={"ID":"16b760c4-0184-4c7b-a1cc-65e0d7841740","Type":"ContainerStarted","Data":"74dd59a8c8a555bb2309ac0d17897076e110bed38352737804f5c909d387f3b5"} Jul 14 21:51:34.155436 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:34.155387 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" podStartSLOduration=9.683232303 podStartE2EDuration="28.155373582s" totalImagesPullingTime="18.472141279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:34.154921882 +0000 UTC m=+29.004391920" watchObservedRunningTime="2026-07-14 21:51:34.155373582 +0000 UTC m=+29.004843621" Jul 14 21:51:34.804342 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:34.804310 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:34.804987 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:34.804425 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:35.112530 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.112440 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:35.112530 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.112521 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:35.113002 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.112564 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:35.127787 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.127763 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:35.128037 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.128002 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:51:35.565159 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.565128 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8dxrp"] Jul 14 21:51:35.565340 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.565234 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:35.565340 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:35.565316 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:35.567728 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.567705 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xpch8"] Jul 14 21:51:35.567880 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.567809 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:35.567950 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:35.567907 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:35.568465 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.568441 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6flwq"] Jul 14 21:51:35.568588 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:35.568533 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:35.568649 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:35.568611 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:36.115913 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:36.115880 2562 generic.go:356] "Generic (PLEG): container finished" podID="535d8ec6-5b58-40eb-b09f-b8b3a8044071" containerID="a33763dc65de35510155170ba57a8225ac2915bf315ccbfa14985c3aad8c13e0" exitCode=0 Jul 14 21:51:36.116398 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:36.115968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" event={"ID":"535d8ec6-5b58-40eb-b09f-b8b3a8044071","Type":"ContainerDied","Data":"a33763dc65de35510155170ba57a8225ac2915bf315ccbfa14985c3aad8c13e0"} Jul 14 21:51:36.805175 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:36.805142 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:36.805368 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:36.805268 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:37.804336 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:37.804122 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:37.804796 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:37.804205 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:37.804796 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:37.804436 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:37.804796 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:37.804456 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:38.122842 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:38.122804 2562 generic.go:356] "Generic (PLEG): container finished" podID="535d8ec6-5b58-40eb-b09f-b8b3a8044071" containerID="7b52de52b53d83a9c8e355d9380950870fece7ddd35f2251403eb29cba08f479" exitCode=0 Jul 14 21:51:38.123013 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:38.122857 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" event={"ID":"535d8ec6-5b58-40eb-b09f-b8b3a8044071","Type":"ContainerDied","Data":"7b52de52b53d83a9c8e355d9380950870fece7ddd35f2251403eb29cba08f479"} Jul 14 21:51:38.430335 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:38.430254 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b4ttn_6212dead-9e15-43dc-bbef-57928fbd39f1/dns-node-resolver/0.log" Jul 14 21:51:38.804421 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:38.804389 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:38.804841 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:38.804504 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:39.001433 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:39.001385 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:39.001604 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:39.001545 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 21:51:39.002046 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:39.002002 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rw49t" Jul 14 21:51:39.599414 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:39.599333 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-frswj_38b6f451-563f-498b-86d9-18dbd2f78850/node-ca/0.log" Jul 14 21:51:39.803921 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:39.803891 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:39.804119 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:39.804037 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpch8" podUID="885409c4-463f-4bc0-a3dc-b9f5a25d37e4" Jul 14 21:51:39.804119 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:39.804080 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:39.804224 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:39.804177 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dxrp" podUID="c7f4d121-c802-43d7-995b-4a54e3d71205" Jul 14 21:51:40.543898 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.543860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:40.544521 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.543973 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcfj\" (UniqueName: \"kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj\") pod \"network-check-target-6flwq\" (UID: \"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70\") " pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:40.544521 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.544005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:40.544521 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:40.544012 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:40.544521 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:40.544091 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret podName:c7f4d121-c802-43d7-995b-4a54e3d71205 nodeName:}" failed. No retries permitted until 2026-07-14 21:52:12.544071733 +0000 UTC m=+67.393541769 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret") pod "global-pull-secret-syncer-8dxrp" (UID: "c7f4d121-c802-43d7-995b-4a54e3d71205") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:51:40.544521 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:40.544116 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:40.544521 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:40.544134 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:51:40.544521 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:40.544154 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs podName:885409c4-463f-4bc0-a3dc-b9f5a25d37e4 nodeName:}" failed. No retries permitted until 2026-07-14 21:52:12.544140263 +0000 UTC m=+67.393610282 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs") pod "network-metrics-daemon-xpch8" (UID: "885409c4-463f-4bc0-a3dc-b9f5a25d37e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:51:40.544521 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:40.544156 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:51:40.544521 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:40.544173 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4kcfj for pod openshift-network-diagnostics/network-check-target-6flwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:40.544521 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:40.544222 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj podName:448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70 nodeName:}" failed. No retries permitted until 2026-07-14 21:52:12.544208299 +0000 UTC m=+67.393678320 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcfj" (UniqueName: "kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj") pod "network-check-target-6flwq" (UID: "448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:51:40.803761 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.803673 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:40.803924 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:40.803820 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6flwq" podUID="448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70" Jul 14 21:51:40.938767 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.938740 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-219.ec2.internal" event="NodeReady" Jul 14 21:51:40.938945 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.938907 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 21:51:40.993289 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.993261 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2krhj"] Jul 14 21:51:40.996677 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.996645 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:40.999433 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.999405 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pkp6j"] Jul 14 21:51:40.999565 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.999502 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 21:51:40.999796 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.999779 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 21:51:40.999886 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:40.999780 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2wg9m\"" Jul 14 21:51:41.001505 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.001489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.003820 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.003796 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 21:51:41.003951 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.003933 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 21:51:41.004035 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.004004 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 21:51:41.004100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.004070 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bh6h4\"" Jul 14 21:51:41.004376 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.004353 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 21:51:41.007601 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.007580 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2krhj"] Jul 14 21:51:41.011630 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.011608 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bnc47"] Jul 14 21:51:41.014624 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.014606 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bnc47" Jul 14 21:51:41.016282 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.016258 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pkp6j"] Jul 14 21:51:41.017422 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.017405 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 21:51:41.017515 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.017438 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f67q8\"" Jul 14 21:51:41.017686 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.017670 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 21:51:41.017813 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.017798 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 21:51:41.024518 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.024499 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bnc47"] Jul 14 21:51:41.148551 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148455 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fbfba53-2715-4194-9075-cf7ff451053e-data-volume\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.148551 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/764d3fb0-f902-4449-b1c7-ae0abdaa8a6d-cert\") pod \"ingress-canary-bnc47\" (UID: \"764d3fb0-f902-4449-b1c7-ae0abdaa8a6d\") " pod="openshift-ingress-canary/ingress-canary-bnc47" Jul 14 21:51:41.148786 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148582 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3380677a-ab5c-4196-864f-891f7cb4a5ed-tmp-dir\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.148786 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fbfba53-2715-4194-9075-cf7ff451053e-crio-socket\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.148786 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148655 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.148786 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rslq\" (UniqueName: \"kubernetes.io/projected/764d3fb0-f902-4449-b1c7-ae0abdaa8a6d-kube-api-access-2rslq\") pod \"ingress-canary-bnc47\" (UID: \"764d3fb0-f902-4449-b1c7-ae0abdaa8a6d\") " pod="openshift-ingress-canary/ingress-canary-bnc47" Jul 14 21:51:41.148786 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148753 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3380677a-ab5c-4196-864f-891f7cb4a5ed-config-volume\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.148786 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktfv2\" (UniqueName: \"kubernetes.io/projected/5fbfba53-2715-4194-9075-cf7ff451053e-kube-api-access-ktfv2\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.149068 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.149068 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148837 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6qd\" (UniqueName: \"kubernetes.io/projected/3380677a-ab5c-4196-864f-891f7cb4a5ed-kube-api-access-gq6qd\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.149068 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.148866 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.249913 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.249880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3380677a-ab5c-4196-864f-891f7cb4a5ed-tmp-dir\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.250112 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.249927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fbfba53-2715-4194-9075-cf7ff451053e-crio-socket\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.250112 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.250112 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rslq\" (UniqueName: \"kubernetes.io/projected/764d3fb0-f902-4449-b1c7-ae0abdaa8a6d-kube-api-access-2rslq\") pod \"ingress-canary-bnc47\" (UID: \"764d3fb0-f902-4449-b1c7-ae0abdaa8a6d\") " pod="openshift-ingress-canary/ingress-canary-bnc47" Jul 14 21:51:41.250271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3380677a-ab5c-4196-864f-891f7cb4a5ed-config-volume\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.250271 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.250152 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:51:41.250271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktfv2\" (UniqueName: \"kubernetes.io/projected/5fbfba53-2715-4194-9075-cf7ff451053e-kube-api-access-ktfv2\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.250271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fbfba53-2715-4194-9075-cf7ff451053e-crio-socket\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.250271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.250271 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.250259 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:51:41.750208431 +0000 UTC m=+36.599678467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : secret "insights-runtime-extractor-tls" not found Jul 14 21:51:41.250574 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250276 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3380677a-ab5c-4196-864f-891f7cb4a5ed-tmp-dir\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.250574 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6qd\" (UniqueName: \"kubernetes.io/projected/3380677a-ab5c-4196-864f-891f7cb4a5ed-kube-api-access-gq6qd\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.250574 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.250574 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250386 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fbfba53-2715-4194-9075-cf7ff451053e-data-volume\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.250574 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/764d3fb0-f902-4449-b1c7-ae0abdaa8a6d-cert\") pod \"ingress-canary-bnc47\" (UID: \"764d3fb0-f902-4449-b1c7-ae0abdaa8a6d\") " pod="openshift-ingress-canary/ingress-canary-bnc47" Jul 14 21:51:41.250574 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.250476 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:51:41.250574 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.250524 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls podName:3380677a-ab5c-4196-864f-891f7cb4a5ed nodeName:}" failed. No retries permitted until 2026-07-14 21:51:41.750507847 +0000 UTC m=+36.599977865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls") pod "dns-default-2krhj" (UID: "3380677a-ab5c-4196-864f-891f7cb4a5ed") : secret "dns-default-metrics-tls" not found Jul 14 21:51:41.250574 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.250557 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 21:51:41.250929 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.250641 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:51:41.750622443 +0000 UTC m=+36.600092482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : configmap "kube-rbac-proxy" not found Jul 14 21:51:41.250929 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250691 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3380677a-ab5c-4196-864f-891f7cb4a5ed-config-volume\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.250929 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.250759 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fbfba53-2715-4194-9075-cf7ff451053e-data-volume\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.255055 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.255010 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/764d3fb0-f902-4449-b1c7-ae0abdaa8a6d-cert\") pod \"ingress-canary-bnc47\" (UID: \"764d3fb0-f902-4449-b1c7-ae0abdaa8a6d\") " pod="openshift-ingress-canary/ingress-canary-bnc47" Jul 14 21:51:41.259624 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.259586 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktfv2\" (UniqueName: \"kubernetes.io/projected/5fbfba53-2715-4194-9075-cf7ff451053e-kube-api-access-ktfv2\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.259736 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.259651 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rslq\" (UniqueName: \"kubernetes.io/projected/764d3fb0-f902-4449-b1c7-ae0abdaa8a6d-kube-api-access-2rslq\") pod \"ingress-canary-bnc47\" (UID: \"764d3fb0-f902-4449-b1c7-ae0abdaa8a6d\") " pod="openshift-ingress-canary/ingress-canary-bnc47" Jul 14 21:51:41.260444 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.260422 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6qd\" (UniqueName: \"kubernetes.io/projected/3380677a-ab5c-4196-864f-891f7cb4a5ed-kube-api-access-gq6qd\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.326437 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.326403 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bnc47" Jul 14 21:51:41.370142 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.370109 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z"] Jul 14 21:51:41.371854 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.371832 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z" Jul 14 21:51:41.374622 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.374598 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:51:41.374622 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.374612 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 14 21:51:41.374879 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.374864 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dwdsl\"" Jul 14 21:51:41.383100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.383075 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z"] Jul 14 21:51:41.554137 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.554092 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz4zt\" (UniqueName: \"kubernetes.io/projected/7e0ae623-c7d3-480b-9f2f-dfe60a436ad2-kube-api-access-kz4zt\") pod \"volume-data-source-validator-78495597f6-q6q9z\" (UID: \"7e0ae623-c7d3-480b-9f2f-dfe60a436ad2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z" Jul 14 21:51:41.655088 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.655054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz4zt\" (UniqueName: \"kubernetes.io/projected/7e0ae623-c7d3-480b-9f2f-dfe60a436ad2-kube-api-access-kz4zt\") pod \"volume-data-source-validator-78495597f6-q6q9z\" (UID: \"7e0ae623-c7d3-480b-9f2f-dfe60a436ad2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z" Jul 14 21:51:41.664175 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.664141 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz4zt\" (UniqueName: \"kubernetes.io/projected/7e0ae623-c7d3-480b-9f2f-dfe60a436ad2-kube-api-access-kz4zt\") pod \"volume-data-source-validator-78495597f6-q6q9z\" (UID: \"7e0ae623-c7d3-480b-9f2f-dfe60a436ad2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z" Jul 14 21:51:41.681927 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.681887 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z" Jul 14 21:51:41.755954 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.755920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.756125 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.755965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:41.756125 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.756082 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 21:51:41.756227 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.756087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:41.756227 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.756154 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:51:42.756133131 +0000 UTC m=+37.605603165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : configmap "kube-rbac-proxy" not found Jul 14 21:51:41.756227 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.756160 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:51:41.756227 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.756154 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:51:41.756427 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.756211 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls podName:3380677a-ab5c-4196-864f-891f7cb4a5ed nodeName:}" failed. No retries permitted until 2026-07-14 21:51:42.756199613 +0000 UTC m=+37.605669630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls") pod "dns-default-2krhj" (UID: "3380677a-ab5c-4196-864f-891f7cb4a5ed") : secret "dns-default-metrics-tls" not found Jul 14 21:51:41.756427 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:41.756307 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:51:42.756283079 +0000 UTC m=+37.605753104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : secret "insights-runtime-extractor-tls" not found Jul 14 21:51:41.804592 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.804510 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:51:41.804592 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.804561 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:51:41.807234 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.807198 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 21:51:41.807367 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.807263 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l5sfp\"" Jul 14 21:51:41.807367 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:41.807323 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 21:51:42.367755 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:42.367573 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bnc47"] Jul 14 21:51:42.373542 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:42.373504 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z"] Jul 14 21:51:42.379518 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:42.379489 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0ae623_c7d3_480b_9f2f_dfe60a436ad2.slice/crio-36ac600eedf17759b6c370547efbee63ee80926f8373e23d7d4095f7d49a29a0 WatchSource:0}: Error finding container 36ac600eedf17759b6c370547efbee63ee80926f8373e23d7d4095f7d49a29a0: Status 404 returned error can't find the container with id 36ac600eedf17759b6c370547efbee63ee80926f8373e23d7d4095f7d49a29a0 Jul 14 21:51:42.380594 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:42.380565 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764d3fb0_f902_4449_b1c7_ae0abdaa8a6d.slice/crio-ff74893c161800ade88cdee23d0140d7316e67538e7d2ff4d07c716e31ddb0a7 WatchSource:0}: Error finding container ff74893c161800ade88cdee23d0140d7316e67538e7d2ff4d07c716e31ddb0a7: Status 404 returned error can't find the container with id ff74893c161800ade88cdee23d0140d7316e67538e7d2ff4d07c716e31ddb0a7 Jul 14 21:51:42.766464 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:42.766370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:42.766464 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:42.766416 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:42.766464 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:42.766460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:42.767008 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:42.766530 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 21:51:42.767008 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:42.766580 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:51:42.767008 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:42.766588 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:51:42.767008 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:42.766618 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:51:44.766593721 +0000 UTC m=+39.616063756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : configmap "kube-rbac-proxy" not found Jul 14 21:51:42.767008 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:42.766638 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:51:44.766629514 +0000 UTC m=+39.616099534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : secret "insights-runtime-extractor-tls" not found Jul 14 21:51:42.767008 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:42.766653 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls podName:3380677a-ab5c-4196-864f-891f7cb4a5ed nodeName:}" failed. No retries permitted until 2026-07-14 21:51:44.766645352 +0000 UTC m=+39.616115370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls") pod "dns-default-2krhj" (UID: "3380677a-ab5c-4196-864f-891f7cb4a5ed") : secret "dns-default-metrics-tls" not found Jul 14 21:51:42.804340 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:42.804312 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:51:42.807014 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:42.806988 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 21:51:42.807014 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:42.806997 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 21:51:42.807927 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:42.807909 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r4xkn\"" Jul 14 21:51:43.138139 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:43.138103 2562 generic.go:356] "Generic (PLEG): container finished" podID="535d8ec6-5b58-40eb-b09f-b8b3a8044071" containerID="945041c0804affe25e0dcfb9150cd7b96741b58f50c30b1110f44af20aec761b" exitCode=0 Jul 14 21:51:43.138299 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:43.138196 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" event={"ID":"535d8ec6-5b58-40eb-b09f-b8b3a8044071","Type":"ContainerDied","Data":"945041c0804affe25e0dcfb9150cd7b96741b58f50c30b1110f44af20aec761b"} Jul 14 21:51:43.139899 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:43.139871 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bnc47" event={"ID":"764d3fb0-f902-4449-b1c7-ae0abdaa8a6d","Type":"ContainerStarted","Data":"ff74893c161800ade88cdee23d0140d7316e67538e7d2ff4d07c716e31ddb0a7"} Jul 14 21:51:43.141292 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:43.141265 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z" event={"ID":"7e0ae623-c7d3-480b-9f2f-dfe60a436ad2","Type":"ContainerStarted","Data":"36ac600eedf17759b6c370547efbee63ee80926f8373e23d7d4095f7d49a29a0"} Jul 14 21:51:44.146971 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:44.146936 2562 generic.go:356] "Generic (PLEG): container finished" podID="535d8ec6-5b58-40eb-b09f-b8b3a8044071" containerID="aed1dc1455699bc2d4157754e3123ca95f7de24b375644221715bc6991457ee7" exitCode=0 Jul 14 21:51:44.147632 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:44.147037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" event={"ID":"535d8ec6-5b58-40eb-b09f-b8b3a8044071","Type":"ContainerDied","Data":"aed1dc1455699bc2d4157754e3123ca95f7de24b375644221715bc6991457ee7"} Jul 14 21:51:44.148821 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:44.148706 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z" event={"ID":"7e0ae623-c7d3-480b-9f2f-dfe60a436ad2","Type":"ContainerStarted","Data":"76839bdd3870a5f0d1983cda38520752fdf13bd8e94556af034c8fc14aca73f0"} Jul 14 21:51:44.188224 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:44.188153 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-q6q9z" podStartSLOduration=1.918292036 podStartE2EDuration="3.188128307s" totalImagesPullingTime="1.269836271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:44.187646742 +0000 UTC m=+39.037116783" watchObservedRunningTime="2026-07-14 21:51:44.188128307 +0000 UTC m=+39.037598347" Jul 14 21:51:44.782936 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:44.782675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:44.783135 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:44.782838 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:51:44.783135 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:44.783090 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 21:51:44.783135 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:44.783105 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:51:48.783087462 +0000 UTC m=+43.632557482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : secret "insights-runtime-extractor-tls" not found Jul 14 21:51:44.783301 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:44.783006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:44.783301 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:44.783146 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:51:48.783129577 +0000 UTC m=+43.632599598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : configmap "kube-rbac-proxy" not found Jul 14 21:51:44.783301 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:44.783236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:44.783440 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:44.783377 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:51:44.783494 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:44.783438 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls podName:3380677a-ab5c-4196-864f-891f7cb4a5ed nodeName:}" failed. No retries permitted until 2026-07-14 21:51:48.783422502 +0000 UTC m=+43.632892523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls") pod "dns-default-2krhj" (UID: "3380677a-ab5c-4196-864f-891f7cb4a5ed") : secret "dns-default-metrics-tls" not found Jul 14 21:51:45.157845 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:45.157762 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" event={"ID":"535d8ec6-5b58-40eb-b09f-b8b3a8044071","Type":"ContainerStarted","Data":"2bff6b664ac6147f1f6e273e68988b0ef298f036336714f90f0e3aab590e0793"} Jul 14 21:51:45.183349 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:45.183302 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cf2lf" podStartSLOduration=10.052824428 podStartE2EDuration="39.183287165s" totalImagesPullingTime="29.130462737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:45.18257783 +0000 UTC m=+40.032047883" watchObservedRunningTime="2026-07-14 21:51:45.183287165 +0000 UTC m=+40.032757204" Jul 14 21:51:46.161790 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:46.161753 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bnc47" event={"ID":"764d3fb0-f902-4449-b1c7-ae0abdaa8a6d","Type":"ContainerStarted","Data":"caa78878b25b62cdb8e5c61b29b764022df0fecfa68ff107cedfe352899f515b"} Jul 14 21:51:46.177949 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:46.177896 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bnc47" podStartSLOduration=3.19115482 podStartE2EDuration="6.177879161s" totalImagesPullingTime="2.986724341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:46.177330898 +0000 UTC m=+41.026800937" watchObservedRunningTime="2026-07-14 21:51:46.177879161 +0000 UTC m=+41.027349200" Jul 14 21:51:47.441296 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.441259 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr"] Jul 14 21:51:47.470489 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.470455 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr"] Jul 14 21:51:47.470643 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.470579 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr" Jul 14 21:51:47.473650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.473626 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 14 21:51:47.473650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.473629 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 14 21:51:47.473819 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.473770 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mbbgf\"" Jul 14 21:51:47.503643 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.503610 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fr8w\" (UniqueName: \"kubernetes.io/projected/ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5-kube-api-access-8fr8w\") pod \"migrator-7f59b54f6c-f4glr\" (UID: \"ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr" Jul 14 21:51:47.604047 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.603992 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fr8w\" (UniqueName: \"kubernetes.io/projected/ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5-kube-api-access-8fr8w\") pod \"migrator-7f59b54f6c-f4glr\" (UID: \"ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr" Jul 14 21:51:47.614173 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.614143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fr8w\" (UniqueName: \"kubernetes.io/projected/ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5-kube-api-access-8fr8w\") pod \"migrator-7f59b54f6c-f4glr\" (UID: \"ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr" Jul 14 21:51:47.779198 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.779163 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr" Jul 14 21:51:47.931436 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:47.928432 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr"] Jul 14 21:51:48.167578 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:48.167484 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr" event={"ID":"ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5","Type":"ContainerStarted","Data":"ee0b7f572cbe32c4bc7b19fcc1f883f553f1638bf641f582b5214acc01c7dc44"} Jul 14 21:51:48.812300 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:48.812243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:48.812300 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:48.812304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:48.812820 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:48.812368 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 21:51:48.812820 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:48.812431 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:51:56.812413593 +0000 UTC m=+51.661883616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : configmap "kube-rbac-proxy" not found Jul 14 21:51:48.812820 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:48.812447 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:51:48.812820 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:48.812368 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:48.812820 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:48.812517 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:51:56.812505058 +0000 UTC m=+51.661975079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : secret "insights-runtime-extractor-tls" not found Jul 14 21:51:48.812820 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:48.812547 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:51:48.812820 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:48.812618 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls podName:3380677a-ab5c-4196-864f-891f7cb4a5ed nodeName:}" failed. No retries permitted until 2026-07-14 21:51:56.812601117 +0000 UTC m=+51.662071153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls") pod "dns-default-2krhj" (UID: "3380677a-ab5c-4196-864f-891f7cb4a5ed") : secret "dns-default-metrics-tls" not found Jul 14 21:51:50.174035 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:50.173996 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr" event={"ID":"ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5","Type":"ContainerStarted","Data":"c8e13f0506e73736be465916dc0f564e228551645b0e85a3584ab97f128fda20"} Jul 14 21:51:50.174506 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:50.174066 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr" event={"ID":"ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5","Type":"ContainerStarted","Data":"599946bfb09d8ea9f8529833daf1f44edf34fa66ed393fa263b0750394f90a68"} Jul 14 21:51:50.189797 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:50.189744 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-f4glr" podStartSLOduration=1.380152561 podStartE2EDuration="3.189727607s" totalImagesPullingTime="1.809575046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:51:50.189314555 +0000 UTC m=+45.038784596" watchObservedRunningTime="2026-07-14 21:51:50.189727607 +0000 UTC m=+45.039197665" Jul 14 21:51:56.865952 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:56.865916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:56.866453 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:56.865993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:56.866453 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:56.866079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:56.866453 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:56.866131 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:51:56.866453 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:51:56.866200 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls podName:5fbfba53-2715-4194-9075-cf7ff451053e nodeName:}" failed. No retries permitted until 2026-07-14 21:52:12.86618202 +0000 UTC m=+67.715652037 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkp6j" (UID: "5fbfba53-2715-4194-9075-cf7ff451053e") : secret "insights-runtime-extractor-tls" not found Jul 14 21:51:56.866634 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:56.866617 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fbfba53-2715-4194-9075-cf7ff451053e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:51:56.868206 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:56.868188 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3380677a-ab5c-4196-864f-891f7cb4a5ed-metrics-tls\") pod \"dns-default-2krhj\" (UID: \"3380677a-ab5c-4196-864f-891f7cb4a5ed\") " pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:56.910118 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:56.910092 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2krhj" Jul 14 21:51:57.047957 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:57.047914 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2krhj"] Jul 14 21:51:57.051017 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:51:57.050988 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3380677a_ab5c_4196_864f_891f7cb4a5ed.slice/crio-973a94b1e17560300e31446c689f2d2f2090fcfcd6aad727889069b0955b5a6c WatchSource:0}: Error finding container 973a94b1e17560300e31446c689f2d2f2090fcfcd6aad727889069b0955b5a6c: Status 404 returned error can't find the container with id 973a94b1e17560300e31446c689f2d2f2090fcfcd6aad727889069b0955b5a6c Jul 14 21:51:57.192413 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:51:57.192326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2krhj" event={"ID":"3380677a-ab5c-4196-864f-891f7cb4a5ed","Type":"ContainerStarted","Data":"973a94b1e17560300e31446c689f2d2f2090fcfcd6aad727889069b0955b5a6c"} Jul 14 21:52:00.203529 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:00.203486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2krhj" event={"ID":"3380677a-ab5c-4196-864f-891f7cb4a5ed","Type":"ContainerStarted","Data":"0300300cf02bfd76002371874710531883eb7cfc616c63bdd6aff660c1176c2a"} Jul 14 21:52:00.203930 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:00.203552 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2krhj" event={"ID":"3380677a-ab5c-4196-864f-891f7cb4a5ed","Type":"ContainerStarted","Data":"6f27d90cc7aaa74676d0dd67a7dfb391eec1681d687c12ee2e336078f415fcc8"} Jul 14 21:52:00.203930 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:00.203592 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2krhj" Jul 14 21:52:00.223135 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:00.223089 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2krhj" podStartSLOduration=18.083620055 podStartE2EDuration="20.223075051s" totalImagesPullingTime="2.139454996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:00.222880341 +0000 UTC m=+55.072350394" watchObservedRunningTime="2026-07-14 21:52:00.223075051 +0000 UTC m=+55.072545067" Jul 14 21:52:07.135876 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:07.135848 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-47d2f" Jul 14 21:52:10.209611 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:10.209582 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2krhj" Jul 14 21:52:12.584824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.584781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcfj\" (UniqueName: \"kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj\") pod \"network-check-target-6flwq\" (UID: \"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70\") " pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:52:12.584824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.584822 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:52:12.585290 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.584952 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:52:12.587692 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.587673 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 21:52:12.588840 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.588825 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 21:52:12.588891 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.588827 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 21:52:12.597274 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.597253 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 21:52:12.597416 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.597398 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7f4d121-c802-43d7-995b-4a54e3d71205-original-pull-secret\") pod \"global-pull-secret-syncer-8dxrp\" (UID: \"c7f4d121-c802-43d7-995b-4a54e3d71205\") " pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:52:12.597474 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.597412 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/885409c4-463f-4bc0-a3dc-b9f5a25d37e4-metrics-certs\") pod \"network-metrics-daemon-xpch8\" (UID: \"885409c4-463f-4bc0-a3dc-b9f5a25d37e4\") " pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:52:12.607314 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.607291 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kcfj\" (UniqueName: \"kubernetes.io/projected/448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70-kube-api-access-4kcfj\") pod \"network-check-target-6flwq\" (UID: \"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70\") " pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:52:12.716744 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.716700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dxrp" Jul 14 21:52:12.724879 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.724859 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l5sfp\"" Jul 14 21:52:12.732944 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.732920 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpch8" Jul 14 21:52:12.821309 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.821256 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r4xkn\"" Jul 14 21:52:12.824529 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.824416 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:52:12.853480 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.853367 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8dxrp"] Jul 14 21:52:12.856647 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:12.856620 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f4d121_c802_43d7_995b_4a54e3d71205.slice/crio-5520c0aec8af2d1a40ae1edf86a8d68bce4941b2b0043b69ed608977891a16c6 WatchSource:0}: Error finding container 5520c0aec8af2d1a40ae1edf86a8d68bce4941b2b0043b69ed608977891a16c6: Status 404 returned error can't find the container with id 5520c0aec8af2d1a40ae1edf86a8d68bce4941b2b0043b69ed608977891a16c6 Jul 14 21:52:12.872949 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.872910 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xpch8"] Jul 14 21:52:12.878060 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:12.878035 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885409c4_463f_4bc0_a3dc_b9f5a25d37e4.slice/crio-eb4ab1ea3214bc5a6722cd9606adddae2cdfd004e5602f7c077b9a1875bac243 WatchSource:0}: Error finding container eb4ab1ea3214bc5a6722cd9606adddae2cdfd004e5602f7c077b9a1875bac243: Status 404 returned error can't find the container with id eb4ab1ea3214bc5a6722cd9606adddae2cdfd004e5602f7c077b9a1875bac243 Jul 14 21:52:12.887815 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.887791 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:52:12.889961 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.889941 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fbfba53-2715-4194-9075-cf7ff451053e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkp6j\" (UID: \"5fbfba53-2715-4194-9075-cf7ff451053e\") " pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:52:12.950366 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:12.950333 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6flwq"] Jul 14 21:52:12.953349 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:12.953315 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448c61ea_7fb3_4c0d_b5fa_302ce0f4ab70.slice/crio-bd0416ad93eb16833aeb6cc180cc98eb47cc4e423b558cbfd0871efa6c02001a WatchSource:0}: Error finding container bd0416ad93eb16833aeb6cc180cc98eb47cc4e423b558cbfd0871efa6c02001a: Status 404 returned error can't find the container with id bd0416ad93eb16833aeb6cc180cc98eb47cc4e423b558cbfd0871efa6c02001a Jul 14 21:52:13.119817 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:13.119734 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bh6h4\"" Jul 14 21:52:13.127824 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:13.127802 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pkp6j" Jul 14 21:52:13.242605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:13.242574 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6flwq" event={"ID":"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70","Type":"ContainerStarted","Data":"bd0416ad93eb16833aeb6cc180cc98eb47cc4e423b558cbfd0871efa6c02001a"} Jul 14 21:52:13.243626 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:13.243602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xpch8" event={"ID":"885409c4-463f-4bc0-a3dc-b9f5a25d37e4","Type":"ContainerStarted","Data":"eb4ab1ea3214bc5a6722cd9606adddae2cdfd004e5602f7c077b9a1875bac243"} Jul 14 21:52:13.244592 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:13.244574 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8dxrp" event={"ID":"c7f4d121-c802-43d7-995b-4a54e3d71205","Type":"ContainerStarted","Data":"5520c0aec8af2d1a40ae1edf86a8d68bce4941b2b0043b69ed608977891a16c6"} Jul 14 21:52:13.249376 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:13.249353 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pkp6j"] Jul 14 21:52:13.251760 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:13.251737 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbfba53_2715_4194_9075_cf7ff451053e.slice/crio-b2fcdfc84694bf4a5630a641bfdfcb0b001181d64b743489d85d8cb9253e98f2 WatchSource:0}: Error finding container b2fcdfc84694bf4a5630a641bfdfcb0b001181d64b743489d85d8cb9253e98f2: Status 404 returned error can't find the container with id b2fcdfc84694bf4a5630a641bfdfcb0b001181d64b743489d85d8cb9253e98f2 Jul 14 21:52:14.251244 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.251153 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkp6j" event={"ID":"5fbfba53-2715-4194-9075-cf7ff451053e","Type":"ContainerStarted","Data":"b2fcdfc84694bf4a5630a641bfdfcb0b001181d64b743489d85d8cb9253e98f2"} Jul 14 21:52:14.750333 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.749524 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s"] Jul 14 21:52:14.763052 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.762599 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-f4fvf"] Jul 14 21:52:14.764849 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.764056 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" Jul 14 21:52:14.768605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.768582 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 14 21:52:14.768880 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.768847 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 14 21:52:14.769613 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.769589 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 14 21:52:14.769850 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.769830 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 14 21:52:14.770502 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.770479 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-r8mdl\"" Jul 14 21:52:14.774990 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.774650 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-f4fvf"] Jul 14 21:52:14.774990 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.774684 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s"] Jul 14 21:52:14.774990 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.774774 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" Jul 14 21:52:14.779506 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.778598 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ndwk5\"" Jul 14 21:52:14.779506 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.778826 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 14 21:52:14.779506 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.779045 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 14 21:52:14.799338 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.799307 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffjt\" (UniqueName: \"kubernetes.io/projected/adeef8a3-993d-411c-93f9-03a2449d5fb8-kube-api-access-cffjt\") pod \"managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s\" (UID: \"adeef8a3-993d-411c-93f9-03a2449d5fb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" Jul 14 21:52:14.799605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.799379 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4481f524-9474-4352-baf0-994335283f40-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-f4fvf\" (UID: \"4481f524-9474-4352-baf0-994335283f40\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" Jul 14 21:52:14.799605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.799408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adeef8a3-993d-411c-93f9-03a2449d5fb8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s\" (UID: \"adeef8a3-993d-411c-93f9-03a2449d5fb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" Jul 14 21:52:14.799605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.799540 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4481f524-9474-4352-baf0-994335283f40-nginx-conf\") pod \"networking-console-plugin-6cf856758c-f4fvf\" (UID: \"4481f524-9474-4352-baf0-994335283f40\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" Jul 14 21:52:14.866745 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.866643 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-jf8ff"] Jul 14 21:52:14.893413 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.893373 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl"] Jul 14 21:52:14.894130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.893763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-jf8ff" Jul 14 21:52:14.902206 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.899814 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 21:52:14.902206 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.899885 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gl67p\"" Jul 14 21:52:14.902206 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.899814 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 21:52:14.902206 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.900449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4481f524-9474-4352-baf0-994335283f40-nginx-conf\") pod \"networking-console-plugin-6cf856758c-f4fvf\" (UID: \"4481f524-9474-4352-baf0-994335283f40\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" Jul 14 21:52:14.902206 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.900485 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffjt\" (UniqueName: \"kubernetes.io/projected/adeef8a3-993d-411c-93f9-03a2449d5fb8-kube-api-access-cffjt\") pod \"managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s\" (UID: \"adeef8a3-993d-411c-93f9-03a2449d5fb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" Jul 14 21:52:14.902206 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.900524 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4481f524-9474-4352-baf0-994335283f40-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-f4fvf\" (UID: \"4481f524-9474-4352-baf0-994335283f40\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" Jul 14 21:52:14.902206 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.900551 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adeef8a3-993d-411c-93f9-03a2449d5fb8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s\" (UID: \"adeef8a3-993d-411c-93f9-03a2449d5fb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" Jul 14 21:52:14.902206 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.901205 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4481f524-9474-4352-baf0-994335283f40-nginx-conf\") pod \"networking-console-plugin-6cf856758c-f4fvf\" (UID: \"4481f524-9474-4352-baf0-994335283f40\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" Jul 14 21:52:14.905556 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.905166 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6557b88bf5-94knb"] Jul 14 21:52:14.905556 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.905253 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adeef8a3-993d-411c-93f9-03a2449d5fb8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s\" (UID: \"adeef8a3-993d-411c-93f9-03a2449d5fb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" Jul 14 21:52:14.906637 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.905878 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl" Jul 14 21:52:14.908784 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.908319 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zltv2\"" Jul 14 21:52:14.908784 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.908622 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 14 21:52:14.911262 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.911215 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4481f524-9474-4352-baf0-994335283f40-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-f4fvf\" (UID: \"4481f524-9474-4352-baf0-994335283f40\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" Jul 14 21:52:14.924980 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.924950 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl"] Jul 14 21:52:14.925120 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.924999 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-jf8ff"] Jul 14 21:52:14.925120 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.925050 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6557b88bf5-94knb"] Jul 14 21:52:14.925120 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.925102 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:14.926675 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.926532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffjt\" (UniqueName: \"kubernetes.io/projected/adeef8a3-993d-411c-93f9-03a2449d5fb8-kube-api-access-cffjt\") pod \"managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s\" (UID: \"adeef8a3-993d-411c-93f9-03a2449d5fb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" Jul 14 21:52:14.928534 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.928106 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 21:52:14.928534 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.928117 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 21:52:14.928534 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.928393 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dhwt6\"" Jul 14 21:52:14.928744 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.928608 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 21:52:14.934018 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.933928 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6557b88bf5-94knb"] Jul 14 21:52:14.936046 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:14.936010 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 21:52:14.952686 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:52:14.952635 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-6p4v2 registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-6p4v2 registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-6557b88bf5-94knb" podUID="61711597-4fbd-471c-ace2-81ddf18b7979" Jul 14 21:52:15.001932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.001695 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-installation-pull-secrets\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.001932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.001751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x64qz\" (UniqueName: \"kubernetes.io/projected/fa555683-9f61-43a9-a9a6-039274be0052-kube-api-access-x64qz\") pod \"downloads-68c9cbb497-jf8ff\" (UID: \"fa555683-9f61-43a9-a9a6-039274be0052\") " pod="openshift-console/downloads-68c9cbb497-jf8ff" Jul 14 21:52:15.001932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.001803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-trusted-ca\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.001932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.001855 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-registry-certificates\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.009114 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.009077 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-bound-sa-token\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.009252 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.009142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-registry-tls\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.009252 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.009182 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4v2\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-kube-api-access-6p4v2\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.009252 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.009220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/823b44cc-8073-4cd0-9698-bf2d1975fccb-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-s8wbl\" (UID: \"823b44cc-8073-4cd0-9698-bf2d1975fccb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl" Jul 14 21:52:15.009407 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.009323 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-image-registry-private-configuration\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.009407 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.009374 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/61711597-4fbd-471c-ace2-81ddf18b7979-ca-trust-extracted\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.089106 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.088775 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" Jul 14 21:52:15.096407 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.096326 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" Jul 14 21:52:15.110734 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.110302 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-trusted-ca\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.110734 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.110350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-registry-certificates\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.110734 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.110390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-bound-sa-token\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.110734 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.110417 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-registry-tls\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.110734 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.110444 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4v2\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-kube-api-access-6p4v2\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.110734 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.110476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/823b44cc-8073-4cd0-9698-bf2d1975fccb-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-s8wbl\" (UID: \"823b44cc-8073-4cd0-9698-bf2d1975fccb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl" Jul 14 21:52:15.110734 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.110522 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-image-registry-private-configuration\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.110734 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.110550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/61711597-4fbd-471c-ace2-81ddf18b7979-ca-trust-extracted\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.110734 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.110583 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-installation-pull-secrets\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.110734 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.110609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64qz\" (UniqueName: \"kubernetes.io/projected/fa555683-9f61-43a9-a9a6-039274be0052-kube-api-access-x64qz\") pod \"downloads-68c9cbb497-jf8ff\" (UID: \"fa555683-9f61-43a9-a9a6-039274be0052\") " pod="openshift-console/downloads-68c9cbb497-jf8ff" Jul 14 21:52:15.111460 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.111327 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-registry-certificates\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.111646 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.111622 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/61711597-4fbd-471c-ace2-81ddf18b7979-ca-trust-extracted\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.112057 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.111991 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-trusted-ca\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.115085 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.114451 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-registry-tls\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.116073 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.116012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/823b44cc-8073-4cd0-9698-bf2d1975fccb-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-s8wbl\" (UID: \"823b44cc-8073-4cd0-9698-bf2d1975fccb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl" Jul 14 21:52:15.116853 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.116817 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-image-registry-private-configuration\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.117096 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.117055 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-installation-pull-secrets\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.125449 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.125410 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64qz\" (UniqueName: \"kubernetes.io/projected/fa555683-9f61-43a9-a9a6-039274be0052-kube-api-access-x64qz\") pod \"downloads-68c9cbb497-jf8ff\" (UID: \"fa555683-9f61-43a9-a9a6-039274be0052\") " pod="openshift-console/downloads-68c9cbb497-jf8ff" Jul 14 21:52:15.127013 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.126915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4v2\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-kube-api-access-6p4v2\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.128589 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.128542 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-bound-sa-token\") pod \"image-registry-6557b88bf5-94knb\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.229458 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.229133 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-jf8ff" Jul 14 21:52:15.245238 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.245203 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl" Jul 14 21:52:15.255932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.255851 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.289177 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.289143 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:15.325825 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.325643 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d5996c69c-zbrhd"] Jul 14 21:52:15.347555 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.347519 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d5996c69c-zbrhd"] Jul 14 21:52:15.347735 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.347651 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.350625 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.350599 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 21:52:15.350769 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.350629 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 21:52:15.351309 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.350876 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 21:52:15.351309 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.350904 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gvsjv\"" Jul 14 21:52:15.351309 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.350980 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 21:52:15.351309 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.351160 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 21:52:15.412799 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.412767 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-registry-tls\") pod \"61711597-4fbd-471c-ace2-81ddf18b7979\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " Jul 14 21:52:15.412969 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.412830 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-registry-certificates\") pod \"61711597-4fbd-471c-ace2-81ddf18b7979\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " Jul 14 21:52:15.412969 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.412868 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-image-registry-private-configuration\") pod \"61711597-4fbd-471c-ace2-81ddf18b7979\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " Jul 14 21:52:15.413104 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413042 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-installation-pull-secrets\") pod \"61711597-4fbd-471c-ace2-81ddf18b7979\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " Jul 14 21:52:15.413104 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413092 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p4v2\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-kube-api-access-6p4v2\") pod \"61711597-4fbd-471c-ace2-81ddf18b7979\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " Jul 14 21:52:15.413203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413130 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-bound-sa-token\") pod \"61711597-4fbd-471c-ace2-81ddf18b7979\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " Jul 14 21:52:15.413203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413169 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/61711597-4fbd-471c-ace2-81ddf18b7979-ca-trust-extracted\") pod \"61711597-4fbd-471c-ace2-81ddf18b7979\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " Jul 14 21:52:15.413299 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413208 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-trusted-ca\") pod \"61711597-4fbd-471c-ace2-81ddf18b7979\" (UID: \"61711597-4fbd-471c-ace2-81ddf18b7979\") " Jul 14 21:52:15.413355 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413312 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-service-ca\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.413412 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-oauth-serving-cert\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.413412 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413398 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-serving-cert\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.413523 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413444 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-oauth-config\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.413523 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-config\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.413523 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.413511 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrq7\" (UniqueName: \"kubernetes.io/projected/4048182e-e81f-4db6-a0f6-2bc42c1e331d-kube-api-access-2hrq7\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.415437 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.415378 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "61711597-4fbd-471c-ace2-81ddf18b7979" (UID: "61711597-4fbd-471c-ace2-81ddf18b7979"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:52:15.416176 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.416131 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61711597-4fbd-471c-ace2-81ddf18b7979-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "61711597-4fbd-471c-ace2-81ddf18b7979" (UID: "61711597-4fbd-471c-ace2-81ddf18b7979"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:52:15.416176 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.416157 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "61711597-4fbd-471c-ace2-81ddf18b7979" (UID: "61711597-4fbd-471c-ace2-81ddf18b7979"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:52:15.419203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.419174 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "61711597-4fbd-471c-ace2-81ddf18b7979" (UID: "61711597-4fbd-471c-ace2-81ddf18b7979"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:52:15.420941 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.420791 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-kube-api-access-6p4v2" (OuterVolumeSpecName: "kube-api-access-6p4v2") pod "61711597-4fbd-471c-ace2-81ddf18b7979" (UID: "61711597-4fbd-471c-ace2-81ddf18b7979"). InnerVolumeSpecName "kube-api-access-6p4v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:52:15.420941 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.420891 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "61711597-4fbd-471c-ace2-81ddf18b7979" (UID: "61711597-4fbd-471c-ace2-81ddf18b7979"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:52:15.422321 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.422166 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "61711597-4fbd-471c-ace2-81ddf18b7979" (UID: "61711597-4fbd-471c-ace2-81ddf18b7979"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:52:15.423205 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.422949 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "61711597-4fbd-471c-ace2-81ddf18b7979" (UID: "61711597-4fbd-471c-ace2-81ddf18b7979"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:52:15.514932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.514797 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-config\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.514932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.514840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrq7\" (UniqueName: \"kubernetes.io/projected/4048182e-e81f-4db6-a0f6-2bc42c1e331d-kube-api-access-2hrq7\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.514932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.514888 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-service-ca\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.514932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.514920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-oauth-serving-cert\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.515242 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.514964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-serving-cert\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.515242 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.514998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-oauth-config\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.515242 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.515063 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-trusted-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:52:15.515242 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.515079 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-registry-tls\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:52:15.515242 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.515094 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/61711597-4fbd-471c-ace2-81ddf18b7979-registry-certificates\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:52:15.515242 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.515109 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-image-registry-private-configuration\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:52:15.515242 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.515130 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/61711597-4fbd-471c-ace2-81ddf18b7979-installation-pull-secrets\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:52:15.515242 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.515144 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p4v2\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-kube-api-access-6p4v2\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:52:15.515242 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.515158 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61711597-4fbd-471c-ace2-81ddf18b7979-bound-sa-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:52:15.515242 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.515175 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/61711597-4fbd-471c-ace2-81ddf18b7979-ca-trust-extracted\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:52:15.516729 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.516700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-service-ca\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.516840 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.516700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-config\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.517102 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.517075 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-oauth-serving-cert\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.518945 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.518893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-oauth-config\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.520406 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.520352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-serving-cert\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.524372 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.524330 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrq7\" (UniqueName: \"kubernetes.io/projected/4048182e-e81f-4db6-a0f6-2bc42c1e331d-kube-api-access-2hrq7\") pod \"console-7d5996c69c-zbrhd\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.578611 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.578575 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s"] Jul 14 21:52:15.593504 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.593422 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-jf8ff"] Jul 14 21:52:15.660958 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.660906 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:15.663975 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:15.663941 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadeef8a3_993d_411c_93f9_03a2449d5fb8.slice/crio-f621e5244651e049a24c924c17163fd6042d05fdfaaa44ae94a02e850fd9bfb0 WatchSource:0}: Error finding container f621e5244651e049a24c924c17163fd6042d05fdfaaa44ae94a02e850fd9bfb0: Status 404 returned error can't find the container with id f621e5244651e049a24c924c17163fd6042d05fdfaaa44ae94a02e850fd9bfb0 Jul 14 21:52:15.664904 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:15.664875 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa555683_9f61_43a9_a9a6_039274be0052.slice/crio-ad955517db1c965c6674e62326f744777e59768141737971543c8af65e899b61 WatchSource:0}: Error finding container ad955517db1c965c6674e62326f744777e59768141737971543c8af65e899b61: Status 404 returned error can't find the container with id ad955517db1c965c6674e62326f744777e59768141737971543c8af65e899b61 Jul 14 21:52:15.823554 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.823521 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-f4fvf"] Jul 14 21:52:15.826572 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:15.826547 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl"] Jul 14 21:52:16.261488 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:16.261392 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-jf8ff" event={"ID":"fa555683-9f61-43a9-a9a6-039274be0052","Type":"ContainerStarted","Data":"ad955517db1c965c6674e62326f744777e59768141737971543c8af65e899b61"} Jul 14 21:52:16.262841 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:16.262816 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6557b88bf5-94knb" Jul 14 21:52:16.263076 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:16.263046 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" event={"ID":"adeef8a3-993d-411c-93f9-03a2449d5fb8","Type":"ContainerStarted","Data":"f621e5244651e049a24c924c17163fd6042d05fdfaaa44ae94a02e850fd9bfb0"} Jul 14 21:52:16.298506 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:16.298470 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6557b88bf5-94knb"] Jul 14 21:52:16.302110 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:16.302067 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6557b88bf5-94knb"] Jul 14 21:52:16.808961 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:16.808923 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61711597-4fbd-471c-ace2-81ddf18b7979" path="/var/lib/kubelet/pods/61711597-4fbd-471c-ace2-81ddf18b7979/volumes" Jul 14 21:52:17.951602 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:17.951556 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4481f524_9474_4352_baf0_994335283f40.slice/crio-3d904899fdf2df691caa3742a1acc2597793ad958d427b8c0439d4f6f5235ad7 WatchSource:0}: Error finding container 3d904899fdf2df691caa3742a1acc2597793ad958d427b8c0439d4f6f5235ad7: Status 404 returned error can't find the container with id 3d904899fdf2df691caa3742a1acc2597793ad958d427b8c0439d4f6f5235ad7 Jul 14 21:52:17.952084 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:17.951823 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823b44cc_8073_4cd0_9698_bf2d1975fccb.slice/crio-cd9a1e334c9a587046ad901ff485f3c2b7035ba3dd0674c4fb3145993241e15f WatchSource:0}: Error finding container cd9a1e334c9a587046ad901ff485f3c2b7035ba3dd0674c4fb3145993241e15f: Status 404 returned error can't find the container with id cd9a1e334c9a587046ad901ff485f3c2b7035ba3dd0674c4fb3145993241e15f Jul 14 21:52:18.273521 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:18.273415 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl" event={"ID":"823b44cc-8073-4cd0-9698-bf2d1975fccb","Type":"ContainerStarted","Data":"cd9a1e334c9a587046ad901ff485f3c2b7035ba3dd0674c4fb3145993241e15f"} Jul 14 21:52:18.275422 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:18.275363 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" event={"ID":"4481f524-9474-4352-baf0-994335283f40","Type":"ContainerStarted","Data":"3d904899fdf2df691caa3742a1acc2597793ad958d427b8c0439d4f6f5235ad7"} Jul 14 21:52:19.906265 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:19.905821 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d5996c69c-zbrhd"] Jul 14 21:52:21.419306 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:21.419268 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4048182e_e81f_4db6_a0f6_2bc42c1e331d.slice/crio-12616457745debb19b3a955a370f85e138b9e5f2328ed5aa29578e67628d80e2 WatchSource:0}: Error finding container 12616457745debb19b3a955a370f85e138b9e5f2328ed5aa29578e67628d80e2: Status 404 returned error can't find the container with id 12616457745debb19b3a955a370f85e138b9e5f2328ed5aa29578e67628d80e2 Jul 14 21:52:22.296479 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:22.296445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5996c69c-zbrhd" event={"ID":"4048182e-e81f-4db6-a0f6-2bc42c1e331d","Type":"ContainerStarted","Data":"12616457745debb19b3a955a370f85e138b9e5f2328ed5aa29578e67628d80e2"} Jul 14 21:52:23.308789 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.308314 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" event={"ID":"4481f524-9474-4352-baf0-994335283f40","Type":"ContainerStarted","Data":"79fff526a8f55922c11453378245e4605991be2783656d8d09e3a7a34458373b"} Jul 14 21:52:23.311558 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.311339 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkp6j" event={"ID":"5fbfba53-2715-4194-9075-cf7ff451053e","Type":"ContainerStarted","Data":"64b6535b18fbda3e14e7a8ec5588e1545d605c75d42f131bd1210604dc25f122"} Jul 14 21:52:23.314877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.314792 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" event={"ID":"adeef8a3-993d-411c-93f9-03a2449d5fb8","Type":"ContainerStarted","Data":"192966f0655910775d83e36921391a985f7fdcb34d6bf42c234461e661dded44"} Jul 14 21:52:23.317862 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.317450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl" event={"ID":"823b44cc-8073-4cd0-9698-bf2d1975fccb","Type":"ContainerStarted","Data":"0f091ddc936f3a049cef298ae3968f7c0aa50c9c38c0d14a45f3c61fed0fddcc"} Jul 14 21:52:23.318214 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.318165 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl" Jul 14 21:52:23.321940 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.321462 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6flwq" event={"ID":"448c61ea-7fb3-4c0d-b5fa-302ce0f4ab70","Type":"ContainerStarted","Data":"d10327c93c7f6642bc2c8412f75956fa6a2cb284dbf5c7610314ce677dad4890"} Jul 14 21:52:23.322078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.321943 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:52:23.325309 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.325263 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xpch8" event={"ID":"885409c4-463f-4bc0-a3dc-b9f5a25d37e4","Type":"ContainerStarted","Data":"b40f5a325630bef0ec741df43706074c5038150c227d18bdcc5a768fea873719"} Jul 14 21:52:23.325931 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.325712 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl" Jul 14 21:52:23.329093 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.328769 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-f4fvf" podStartSLOduration=4.255680939 podStartE2EDuration="9.32875276s" totalImagesPullingTime="5.073071821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:23.325793329 +0000 UTC m=+78.175263371" watchObservedRunningTime="2026-07-14 21:52:23.32875276 +0000 UTC m=+78.178222800" Jul 14 21:52:23.329901 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.329882 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8dxrp" event={"ID":"c7f4d121-c802-43d7-995b-4a54e3d71205","Type":"ContainerStarted","Data":"068a24b6c6e80cb1d6906c3b7137d7aae8f18949b0ec7822e4761fbda46607d9"} Jul 14 21:52:23.367226 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.367172 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-s8wbl" podStartSLOduration=4.2849638930000005 podStartE2EDuration="9.367152914s" totalImagesPullingTime="5.082189021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:23.365937511 +0000 UTC m=+78.215407561" watchObservedRunningTime="2026-07-14 21:52:23.367152914 +0000 UTC m=+78.216622959" Jul 14 21:52:23.367467 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.367445 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65d99d5b9d-qqn4s" podStartSLOduration=2.026931381 podStartE2EDuration="9.367439187s" totalImagesPullingTime="7.340507806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:23.344322517 +0000 UTC m=+78.193792557" watchObservedRunningTime="2026-07-14 21:52:23.367439187 +0000 UTC m=+78.216909226" Jul 14 21:52:23.401629 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.400929 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xpch8" podStartSLOduration=69.563727095 podStartE2EDuration="1m16.400906301s" totalImagesPullingTime="6.837179206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:23.39755696 +0000 UTC m=+78.247027002" watchObservedRunningTime="2026-07-14 21:52:23.400906301 +0000 UTC m=+78.250376345" Jul 14 21:52:23.432209 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.432154 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6flwq" podStartSLOduration=67.742839078 podStartE2EDuration="1m17.43213202s" totalImagesPullingTime="9.689292942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:23.430878031 +0000 UTC m=+78.280348070" watchObservedRunningTime="2026-07-14 21:52:23.43213202 +0000 UTC m=+78.281602060" Jul 14 21:52:23.460727 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:23.460501 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8dxrp" podStartSLOduration=66.350317031 podStartE2EDuration="1m16.460478093s" totalImagesPullingTime="10.110161062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:23.459607125 +0000 UTC m=+78.309077176" watchObservedRunningTime="2026-07-14 21:52:23.460478093 +0000 UTC m=+78.309948132" Jul 14 21:52:24.113093 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.112948 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86f44f987c-dmdmh"] Jul 14 21:52:24.116518 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.116491 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.124839 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.124803 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 21:52:24.130792 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.130761 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f44f987c-dmdmh"] Jul 14 21:52:24.183701 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.183085 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-mnwlg"] Jul 14 21:52:24.183701 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.183571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-trusted-ca-bundle\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.183701 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.183622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4d4\" (UniqueName: \"kubernetes.io/projected/33a1ec5b-07ec-4853-bb95-1bf77802e511-kube-api-access-pl4d4\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.183990 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.183720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-serving-cert\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.183990 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.183754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-config\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.183990 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.183797 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-oauth-config\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.183990 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.183824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-oauth-serving-cert\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.183990 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.183864 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-service-ca\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.186571 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.186551 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.189489 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.189330 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 14 21:52:24.189489 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.189341 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9sjzf\"" Jul 14 21:52:24.189489 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.189373 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 21:52:24.189714 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.189671 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 21:52:24.189714 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.189694 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 14 21:52:24.189817 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.189773 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 21:52:24.200746 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.200678 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-mnwlg"] Jul 14 21:52:24.285199 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-oauth-config\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.285381 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285210 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c620f4d1-b9ca-456e-abd3-a6982435ca5e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.285381 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285241 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-oauth-serving-cert\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.285381 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-service-ca\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.285381 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllzg\" (UniqueName: \"kubernetes.io/projected/c620f4d1-b9ca-456e-abd3-a6982435ca5e-kube-api-access-fllzg\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.285574 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285391 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-trusted-ca-bundle\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.285574 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c620f4d1-b9ca-456e-abd3-a6982435ca5e-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.285574 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285512 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4d4\" (UniqueName: \"kubernetes.io/projected/33a1ec5b-07ec-4853-bb95-1bf77802e511-kube-api-access-pl4d4\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.285730 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285625 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c620f4d1-b9ca-456e-abd3-a6982435ca5e-metrics-client-ca\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.285730 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-serving-cert\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.285730 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.285702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-config\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.286164 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.286131 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-service-ca\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.286290 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.286133 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-oauth-serving-cert\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.286354 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.286291 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-trusted-ca-bundle\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.286413 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.286372 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-config\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.287936 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.287917 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-oauth-config\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.288356 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.288333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-serving-cert\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.295107 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.295082 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4d4\" (UniqueName: \"kubernetes.io/projected/33a1ec5b-07ec-4853-bb95-1bf77802e511-kube-api-access-pl4d4\") pod \"console-86f44f987c-dmdmh\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.342941 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.342896 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xpch8" event={"ID":"885409c4-463f-4bc0-a3dc-b9f5a25d37e4","Type":"ContainerStarted","Data":"bfff79ac32b0036e6fb94a81898880fbc6c5754854753c4e8329647c5f13dbd8"} Jul 14 21:52:24.346280 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.346247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkp6j" event={"ID":"5fbfba53-2715-4194-9075-cf7ff451053e","Type":"ContainerStarted","Data":"4552c42ab33c31d24ac517911ec58ceced07944d5ac7980e33de7f691babe1a7"} Jul 14 21:52:24.387612 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.386762 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllzg\" (UniqueName: \"kubernetes.io/projected/c620f4d1-b9ca-456e-abd3-a6982435ca5e-kube-api-access-fllzg\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.387612 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.387005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c620f4d1-b9ca-456e-abd3-a6982435ca5e-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.387612 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.387069 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c620f4d1-b9ca-456e-abd3-a6982435ca5e-metrics-client-ca\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.387612 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.387187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c620f4d1-b9ca-456e-abd3-a6982435ca5e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.387954 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.387787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c620f4d1-b9ca-456e-abd3-a6982435ca5e-metrics-client-ca\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.390996 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.390948 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c620f4d1-b9ca-456e-abd3-a6982435ca5e-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.391156 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.391002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c620f4d1-b9ca-456e-abd3-a6982435ca5e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.397977 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.397928 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllzg\" (UniqueName: \"kubernetes.io/projected/c620f4d1-b9ca-456e-abd3-a6982435ca5e-kube-api-access-fllzg\") pod \"prometheus-operator-64785b6675-mnwlg\" (UID: \"c620f4d1-b9ca-456e-abd3-a6982435ca5e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:24.429110 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.428761 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:24.499329 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:24.499289 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" Jul 14 21:52:27.245078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:27.244786 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-mnwlg"] Jul 14 21:52:27.247350 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:27.247319 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc620f4d1_b9ca_456e_abd3_a6982435ca5e.slice/crio-7d2a170cc0788c0ab7c6c0afefb7a0fc59b4101b1f1466f3eb857c994036220e WatchSource:0}: Error finding container 7d2a170cc0788c0ab7c6c0afefb7a0fc59b4101b1f1466f3eb857c994036220e: Status 404 returned error can't find the container with id 7d2a170cc0788c0ab7c6c0afefb7a0fc59b4101b1f1466f3eb857c994036220e Jul 14 21:52:27.265308 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:27.265271 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f44f987c-dmdmh"] Jul 14 21:52:27.268059 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:27.268014 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a1ec5b_07ec_4853_bb95_1bf77802e511.slice/crio-0033addc5989d67d89a52f22c0335a9ff19c36474bc753bf8bc5c9b3afd03304 WatchSource:0}: Error finding container 0033addc5989d67d89a52f22c0335a9ff19c36474bc753bf8bc5c9b3afd03304: Status 404 returned error can't find the container with id 0033addc5989d67d89a52f22c0335a9ff19c36474bc753bf8bc5c9b3afd03304 Jul 14 21:52:27.368175 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:27.368140 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f44f987c-dmdmh" event={"ID":"33a1ec5b-07ec-4853-bb95-1bf77802e511","Type":"ContainerStarted","Data":"61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8"} Jul 14 21:52:27.368472 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:27.368454 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f44f987c-dmdmh" event={"ID":"33a1ec5b-07ec-4853-bb95-1bf77802e511","Type":"ContainerStarted","Data":"0033addc5989d67d89a52f22c0335a9ff19c36474bc753bf8bc5c9b3afd03304"} Jul 14 21:52:27.370979 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:27.370950 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5996c69c-zbrhd" event={"ID":"4048182e-e81f-4db6-a0f6-2bc42c1e331d","Type":"ContainerStarted","Data":"74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a"} Jul 14 21:52:27.373407 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:27.373380 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" event={"ID":"c620f4d1-b9ca-456e-abd3-a6982435ca5e","Type":"ContainerStarted","Data":"7d2a170cc0788c0ab7c6c0afefb7a0fc59b4101b1f1466f3eb857c994036220e"} Jul 14 21:52:27.377100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:27.377071 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkp6j" event={"ID":"5fbfba53-2715-4194-9075-cf7ff451053e","Type":"ContainerStarted","Data":"baa0389da516b3d800e4b596cfe3cc506a6f0b34fade5d24eb306f6e2ed5f92a"} Jul 14 21:52:27.390106 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:27.390007 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86f44f987c-dmdmh" podStartSLOduration=3.389988416 podStartE2EDuration="3.389988416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:27.388531875 +0000 UTC m=+82.238001913" watchObservedRunningTime="2026-07-14 21:52:27.389988416 +0000 UTC m=+82.239458454" Jul 14 21:52:27.407597 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:27.407544 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d5996c69c-zbrhd" podStartSLOduration=6.759817438 podStartE2EDuration="12.407525289s" totalImagesPullingTime="5.647707851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:27.406241122 +0000 UTC m=+82.255711161" watchObservedRunningTime="2026-07-14 21:52:27.407525289 +0000 UTC m=+82.256995329" Jul 14 21:52:27.426227 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:27.426153 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pkp6j" podStartSLOduration=33.820914466 podStartE2EDuration="47.426136456s" totalImagesPullingTime="13.60522199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:51:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:27.425102811 +0000 UTC m=+82.274572875" watchObservedRunningTime="2026-07-14 21:52:27.426136456 +0000 UTC m=+82.275606494" Jul 14 21:52:34.429770 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:34.429733 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:34.430302 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:34.429817 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:34.435098 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:34.435077 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:35.412632 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:35.412603 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:52:35.464140 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:35.464105 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d5996c69c-zbrhd"] Jul 14 21:52:35.661356 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:35.661327 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:52:36.415365 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:36.415323 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-jf8ff" event={"ID":"fa555683-9f61-43a9-a9a6-039274be0052","Type":"ContainerStarted","Data":"d8210f45d95d076b6e5dd0f1e6d162bcd830442841da23c74062fecc6f7933d2"} Jul 14 21:52:36.415737 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:36.415689 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-jf8ff" Jul 14 21:52:36.417776 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:36.417748 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" event={"ID":"c620f4d1-b9ca-456e-abd3-a6982435ca5e","Type":"ContainerStarted","Data":"ca2182beac41e632d96472f2a430bcfa59c3b61d7191b43b567f679cf461b59f"} Jul 14 21:52:36.417892 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:36.417813 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" event={"ID":"c620f4d1-b9ca-456e-abd3-a6982435ca5e","Type":"ContainerStarted","Data":"86ac80e185d6d8e8ca3487068f54ddf65dc3a3b20681abd23b84398ce9c27abc"} Jul 14 21:52:36.429245 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:36.429216 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-jf8ff" Jul 14 21:52:36.440542 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:36.440458 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-jf8ff" podStartSLOduration=2.084371461 podStartE2EDuration="22.440437099s" totalImagesPullingTime="20.356065638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:36.439144873 +0000 UTC m=+91.288614911" watchObservedRunningTime="2026-07-14 21:52:36.440437099 +0000 UTC m=+91.289907138" Jul 14 21:52:36.460793 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:36.460731 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-mnwlg" podStartSLOduration=3.725826857 podStartE2EDuration="12.460708386s" totalImagesPullingTime="8.734881529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:36.459619202 +0000 UTC m=+91.309089242" watchObservedRunningTime="2026-07-14 21:52:36.460708386 +0000 UTC m=+91.310178427" Jul 14 21:52:38.655475 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.655428 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5f67q"] Jul 14 21:52:38.687448 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.687409 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5f67q"] Jul 14 21:52:38.687645 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.687478 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4xhpc"] Jul 14 21:52:38.687718 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.687643 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.690675 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.690646 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 14 21:52:38.690969 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.690945 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tsdp7\"" Jul 14 21:52:38.691360 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.691206 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 14 21:52:38.691360 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.691282 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 14 21:52:38.709551 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.709528 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.712644 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.712619 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 21:52:38.712773 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.712619 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vxhfs\"" Jul 14 21:52:38.712773 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.712691 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 21:52:38.712980 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.712962 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 21:52:38.806953 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.806913 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/104b706a-2784-45de-8889-3b9469d51c72-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.807159 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.806961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-accelerators-collector-config\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.807159 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/104b706a-2784-45de-8889-3b9469d51c72-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.807159 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-textfile\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.807159 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/104b706a-2784-45de-8889-3b9469d51c72-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.807159 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/104b706a-2784-45de-8889-3b9469d51c72-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.807472 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/104b706a-2784-45de-8889-3b9469d51c72-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.807472 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6656125-2156-4760-9a62-a7713a16e946-root\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.807472 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6656125-2156-4760-9a62-a7713a16e946-metrics-client-ca\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.807472 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqlh\" (UniqueName: \"kubernetes.io/projected/104b706a-2784-45de-8889-3b9469d51c72-kube-api-access-rhqlh\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.807472 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807355 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6656125-2156-4760-9a62-a7713a16e946-sys\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.807472 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-wtmp\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.807472 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-tls\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.807472 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807448 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.807828 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.807477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb5hb\" (UniqueName: \"kubernetes.io/projected/f6656125-2156-4760-9a62-a7713a16e946-kube-api-access-nb5hb\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.908425 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/104b706a-2784-45de-8889-3b9469d51c72-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.908425 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908397 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-textfile\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.908650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/104b706a-2784-45de-8889-3b9469d51c72-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.908650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/104b706a-2784-45de-8889-3b9469d51c72-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.908650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/104b706a-2784-45de-8889-3b9469d51c72-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.908650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6656125-2156-4760-9a62-a7713a16e946-root\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.908650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6656125-2156-4760-9a62-a7713a16e946-metrics-client-ca\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.908650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908574 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqlh\" (UniqueName: \"kubernetes.io/projected/104b706a-2784-45de-8889-3b9469d51c72-kube-api-access-rhqlh\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.908650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908596 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6656125-2156-4760-9a62-a7713a16e946-sys\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.908650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908622 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-wtmp\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.908650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-tls\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.909110 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908673 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.909110 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb5hb\" (UniqueName: \"kubernetes.io/projected/f6656125-2156-4760-9a62-a7713a16e946-kube-api-access-nb5hb\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.909110 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/104b706a-2784-45de-8889-3b9469d51c72-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.909110 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908783 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/104b706a-2784-45de-8889-3b9469d51c72-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.909110 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908793 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-textfile\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.909110 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908788 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-accelerators-collector-config\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.909110 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.908871 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6656125-2156-4760-9a62-a7713a16e946-sys\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.909435 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.909195 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/104b706a-2784-45de-8889-3b9469d51c72-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.909435 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.909271 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6656125-2156-4760-9a62-a7713a16e946-root\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.909435 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.909280 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-accelerators-collector-config\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.909435 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.909397 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-wtmp\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.909752 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.909724 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6656125-2156-4760-9a62-a7713a16e946-metrics-client-ca\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.909829 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:52:38.909818 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 21:52:38.909886 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:52:38.909859 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-tls podName:f6656125-2156-4760-9a62-a7713a16e946 nodeName:}" failed. No retries permitted until 2026-07-14 21:52:39.40984678 +0000 UTC m=+94.259316797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-tls") pod "node-exporter-4xhpc" (UID: "f6656125-2156-4760-9a62-a7713a16e946") : secret "node-exporter-tls" not found Jul 14 21:52:38.910008 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.909980 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/104b706a-2784-45de-8889-3b9469d51c72-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.911740 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.911716 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.911848 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.911780 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/104b706a-2784-45de-8889-3b9469d51c72-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.912145 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.912122 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/104b706a-2784-45de-8889-3b9469d51c72-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.918688 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.918665 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqlh\" (UniqueName: \"kubernetes.io/projected/104b706a-2784-45de-8889-3b9469d51c72-kube-api-access-rhqlh\") pod \"kube-state-metrics-785778ff74-5f67q\" (UID: \"104b706a-2784-45de-8889-3b9469d51c72\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:38.918793 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.918754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb5hb\" (UniqueName: \"kubernetes.io/projected/f6656125-2156-4760-9a62-a7713a16e946-kube-api-access-nb5hb\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:38.999830 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:38.999796 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" Jul 14 21:52:39.179437 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.179405 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5f67q"] Jul 14 21:52:39.182228 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:39.182185 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104b706a_2784_45de_8889_3b9469d51c72.slice/crio-b9f0d2d8dd7563075050e6cd6234747ba4231f5402939e008b76e59bda7ef9eb WatchSource:0}: Error finding container b9f0d2d8dd7563075050e6cd6234747ba4231f5402939e008b76e59bda7ef9eb: Status 404 returned error can't find the container with id b9f0d2d8dd7563075050e6cd6234747ba4231f5402939e008b76e59bda7ef9eb Jul 14 21:52:39.413650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.413612 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-tls\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:39.416281 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.416252 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6656125-2156-4760-9a62-a7713a16e946-node-exporter-tls\") pod \"node-exporter-4xhpc\" (UID: \"f6656125-2156-4760-9a62-a7713a16e946\") " pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:39.431064 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.430966 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" event={"ID":"104b706a-2784-45de-8889-3b9469d51c72","Type":"ContainerStarted","Data":"b9f0d2d8dd7563075050e6cd6234747ba4231f5402939e008b76e59bda7ef9eb"} Jul 14 21:52:39.620283 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.620246 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4xhpc" Jul 14 21:52:39.652809 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:39.652771 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6656125_2156_4760_9a62_a7713a16e946.slice/crio-8066a8807d43055d503c3acd47c5407dbfd4e88ca9003606f65dc8277a63d8e4 WatchSource:0}: Error finding container 8066a8807d43055d503c3acd47c5407dbfd4e88ca9003606f65dc8277a63d8e4: Status 404 returned error can't find the container with id 8066a8807d43055d503c3acd47c5407dbfd4e88ca9003606f65dc8277a63d8e4 Jul 14 21:52:39.659519 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.659491 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 21:52:39.689685 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.689609 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 21:52:39.689830 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.689754 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.696176 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.695862 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 14 21:52:39.696176 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.695892 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 14 21:52:39.696386 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.696300 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 14 21:52:39.696386 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.696361 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 14 21:52:39.697378 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.697098 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 14 21:52:39.697378 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.697115 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 14 21:52:39.697378 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.697254 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 14 21:52:39.697613 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.697398 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 14 21:52:39.697613 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.697415 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n5q5n\"" Jul 14 21:52:39.697613 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.697585 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 14 21:52:39.816989 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.816952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817176 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.816995 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817176 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksshl\" (UniqueName: \"kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-kube-api-access-ksshl\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817176 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817110 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817176 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817154 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817341 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-web-config\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817341 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817221 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817341 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817283 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-config-volume\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817469 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817469 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817372 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817469 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817443 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817584 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-config-out\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.817584 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.817552 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919039 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.918927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919039 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.918976 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919039 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919039 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-config-out\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919403 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919403 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919403 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919146 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919403 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919171 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksshl\" (UniqueName: \"kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-kube-api-access-ksshl\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919403 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919210 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919403 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919403 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-web-config\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919403 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919293 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919403 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919323 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-config-volume\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.919858 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.919673 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.926537 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.924932 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-config-out\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.926537 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.925910 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.926537 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.925974 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.926537 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.926475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.926832 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.926588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.931770 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.931475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.932904 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.932152 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-web-config\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.932904 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.932209 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.932904 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.932478 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.932904 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.932529 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.932904 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.932860 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-config-volume\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:39.935189 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:39.935151 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksshl\" (UniqueName: \"kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-kube-api-access-ksshl\") pod \"alertmanager-main-0\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:40.003098 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:40.003007 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:52:40.214585 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:40.214542 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 21:52:40.219469 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:40.219415 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ccd7939_3a63_458c_b21c_9236c7228b29.slice/crio-f5f87361e2c82a9da48edeea962b9bd87abc7296e2d594187add31786deb3304 WatchSource:0}: Error finding container f5f87361e2c82a9da48edeea962b9bd87abc7296e2d594187add31786deb3304: Status 404 returned error can't find the container with id f5f87361e2c82a9da48edeea962b9bd87abc7296e2d594187add31786deb3304 Jul 14 21:52:40.440716 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:40.440673 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerStarted","Data":"f5f87361e2c82a9da48edeea962b9bd87abc7296e2d594187add31786deb3304"} Jul 14 21:52:40.443539 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:40.443498 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4xhpc" event={"ID":"f6656125-2156-4760-9a62-a7713a16e946","Type":"ContainerStarted","Data":"8066a8807d43055d503c3acd47c5407dbfd4e88ca9003606f65dc8277a63d8e4"} Jul 14 21:52:42.455095 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:42.455057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" event={"ID":"104b706a-2784-45de-8889-3b9469d51c72","Type":"ContainerStarted","Data":"95fe87b63104cf8a43d7bfde8901104d5dee534ead5636bf693c3b7049bcf52a"} Jul 14 21:52:42.455609 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:42.455170 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" event={"ID":"104b706a-2784-45de-8889-3b9469d51c72","Type":"ContainerStarted","Data":"5d44ac28ca96ecb7328299a40d91093d876006f4fcaccfe8a0403198bf97c0e5"} Jul 14 21:52:42.455609 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:42.455206 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" event={"ID":"104b706a-2784-45de-8889-3b9469d51c72","Type":"ContainerStarted","Data":"2702702d9c8a4297360834878606832ad1e5ab0a418ebbd00962f4a66700412d"} Jul 14 21:52:42.456981 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:42.456952 2562 generic.go:356] "Generic (PLEG): container finished" podID="f6656125-2156-4760-9a62-a7713a16e946" containerID="8845953ae04c87bd0a30bc6ce9aae1cca29465b36c07d4c1d980d557ec68e192" exitCode=0 Jul 14 21:52:42.457140 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:42.457017 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4xhpc" event={"ID":"f6656125-2156-4760-9a62-a7713a16e946","Type":"ContainerDied","Data":"8845953ae04c87bd0a30bc6ce9aae1cca29465b36c07d4c1d980d557ec68e192"} Jul 14 21:52:42.495883 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:42.495824 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-5f67q" podStartSLOduration=2.071938115 podStartE2EDuration="4.495802496s" totalImagesPullingTime="2.423864381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:42.474781107 +0000 UTC m=+97.324251147" watchObservedRunningTime="2026-07-14 21:52:42.495802496 +0000 UTC m=+97.345272550" Jul 14 21:52:43.337183 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.337144 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-684f8fd54b-l6wt5"] Jul 14 21:52:43.369690 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.369654 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq"] Jul 14 21:52:43.369850 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.369796 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.393397 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.393366 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-684f8fd54b-l6wt5"] Jul 14 21:52:43.393557 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.393410 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq"] Jul 14 21:52:43.393557 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.393478 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:43.400595 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.400565 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 14 21:52:43.400768 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.400609 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hcwzc\"" Jul 14 21:52:43.453657 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.453618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-oauth-serving-cert\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.453835 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.453662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-service-ca\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.453835 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.453736 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-console-config\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.453835 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.453796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn85c\" (UniqueName: \"kubernetes.io/projected/df4bd63a-9145-4a72-bace-902d06ddcae5-kube-api-access-hn85c\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.453835 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.453833 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-trusted-ca-bundle\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.454060 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.453865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-oauth-config\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.454060 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.453921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-serving-cert\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.464449 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.464414 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4xhpc" event={"ID":"f6656125-2156-4760-9a62-a7713a16e946","Type":"ContainerStarted","Data":"b6f7ae011137997114d50fab6b3bdb9b165a3ec166f096551725dcf91c3bae78"} Jul 14 21:52:43.464876 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.464533 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4xhpc" event={"ID":"f6656125-2156-4760-9a62-a7713a16e946","Type":"ContainerStarted","Data":"cfc3d64cc211400d25ac8d9c46d2c22ec5078ad81836c99a61bc16fe2015e0b9"} Jul 14 21:52:43.466382 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.466360 2562 generic.go:356] "Generic (PLEG): container finished" podID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerID="0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930" exitCode=0 Jul 14 21:52:43.466516 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.466448 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerDied","Data":"0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930"} Jul 14 21:52:43.513494 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.513434 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4xhpc" podStartSLOduration=3.407300539 podStartE2EDuration="5.513407046s" totalImagesPullingTime="1.956264219s" totalInitContainerRuntime="149.842288ms" isStatefulPod=true podCreationTimestamp="2026-07-14 21:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:43.512681969 +0000 UTC m=+98.362152010" watchObservedRunningTime="2026-07-14 21:52:43.513407046 +0000 UTC m=+98.362877087" Jul 14 21:52:43.555087 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.555018 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-serving-cert\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.555257 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.555103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07efdcff-0877-4700-83cd-441042289f78-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-b77hq\" (UID: \"07efdcff-0877-4700-83cd-441042289f78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:43.556078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.555678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-oauth-serving-cert\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.556078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.555723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qp57\" (UniqueName: \"kubernetes.io/projected/07efdcff-0877-4700-83cd-441042289f78-kube-api-access-8qp57\") pod \"monitoring-plugin-5f68df7f58-b77hq\" (UID: \"07efdcff-0877-4700-83cd-441042289f78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:43.556078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.555774 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-service-ca\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.556078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.555833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-console-config\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.556078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.555899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn85c\" (UniqueName: \"kubernetes.io/projected/df4bd63a-9145-4a72-bace-902d06ddcae5-kube-api-access-hn85c\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.556078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.555944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-trusted-ca-bundle\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.556078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.555986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-oauth-config\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.556899 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.556668 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-console-config\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.557008 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.556973 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-service-ca\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.557516 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.557475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-oauth-serving-cert\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.557763 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.557734 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-trusted-ca-bundle\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.558249 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.558223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-serving-cert\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.559263 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.559243 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-oauth-config\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.584759 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.584724 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn85c\" (UniqueName: \"kubernetes.io/projected/df4bd63a-9145-4a72-bace-902d06ddcae5-kube-api-access-hn85c\") pod \"console-684f8fd54b-l6wt5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.657698 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.657598 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qp57\" (UniqueName: \"kubernetes.io/projected/07efdcff-0877-4700-83cd-441042289f78-kube-api-access-8qp57\") pod \"monitoring-plugin-5f68df7f58-b77hq\" (UID: \"07efdcff-0877-4700-83cd-441042289f78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:43.657698 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.657681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07efdcff-0877-4700-83cd-441042289f78-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-b77hq\" (UID: \"07efdcff-0877-4700-83cd-441042289f78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:43.657923 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:52:43.657845 2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 14 21:52:43.658372 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:52:43.658330 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07efdcff-0877-4700-83cd-441042289f78-monitoring-plugin-cert podName:07efdcff-0877-4700-83cd-441042289f78 nodeName:}" failed. No retries permitted until 2026-07-14 21:52:44.158299672 +0000 UTC m=+99.007769717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/07efdcff-0877-4700-83cd-441042289f78-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-b77hq" (UID: "07efdcff-0877-4700-83cd-441042289f78") : secret "monitoring-plugin-cert" not found Jul 14 21:52:43.669061 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.669011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qp57\" (UniqueName: \"kubernetes.io/projected/07efdcff-0877-4700-83cd-441042289f78-kube-api-access-8qp57\") pod \"monitoring-plugin-5f68df7f58-b77hq\" (UID: \"07efdcff-0877-4700-83cd-441042289f78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:43.683746 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.683708 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:43.890576 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:43.890537 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-684f8fd54b-l6wt5"] Jul 14 21:52:43.895129 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:43.895103 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4bd63a_9145_4a72_bace_902d06ddcae5.slice/crio-3aed4e327026ab4c77ec3e55d2c2c3518b15ffc34b6c1540fe7d94824de3547f WatchSource:0}: Error finding container 3aed4e327026ab4c77ec3e55d2c2c3518b15ffc34b6c1540fe7d94824de3547f: Status 404 returned error can't find the container with id 3aed4e327026ab4c77ec3e55d2c2c3518b15ffc34b6c1540fe7d94824de3547f Jul 14 21:52:44.163033 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:44.162977 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07efdcff-0877-4700-83cd-441042289f78-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-b77hq\" (UID: \"07efdcff-0877-4700-83cd-441042289f78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:44.166489 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:44.166455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07efdcff-0877-4700-83cd-441042289f78-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-b77hq\" (UID: \"07efdcff-0877-4700-83cd-441042289f78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:44.305279 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:44.305043 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:44.474434 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:44.474396 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684f8fd54b-l6wt5" event={"ID":"df4bd63a-9145-4a72-bace-902d06ddcae5","Type":"ContainerStarted","Data":"07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7"} Jul 14 21:52:44.474871 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:44.474453 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684f8fd54b-l6wt5" event={"ID":"df4bd63a-9145-4a72-bace-902d06ddcae5","Type":"ContainerStarted","Data":"3aed4e327026ab4c77ec3e55d2c2c3518b15ffc34b6c1540fe7d94824de3547f"} Jul 14 21:52:44.495690 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:44.495632 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-684f8fd54b-l6wt5" podStartSLOduration=1.495611662 podStartE2EDuration="1.495611662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:44.494320687 +0000 UTC m=+99.343790726" watchObservedRunningTime="2026-07-14 21:52:44.495611662 +0000 UTC m=+99.345081701" Jul 14 21:52:44.500150 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:44.500115 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq"] Jul 14 21:52:44.504305 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:44.504281 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07efdcff_0877_4700_83cd_441042289f78.slice/crio-d43677a0addc1d3070fc97d80855268fc3bd29a7e6dd529929d5d1f8b85fc2a3 WatchSource:0}: Error finding container d43677a0addc1d3070fc97d80855268fc3bd29a7e6dd529929d5d1f8b85fc2a3: Status 404 returned error can't find the container with id d43677a0addc1d3070fc97d80855268fc3bd29a7e6dd529929d5d1f8b85fc2a3 Jul 14 21:52:45.482443 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:45.482395 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" event={"ID":"07efdcff-0877-4700-83cd-441042289f78","Type":"ContainerStarted","Data":"d43677a0addc1d3070fc97d80855268fc3bd29a7e6dd529929d5d1f8b85fc2a3"} Jul 14 21:52:46.488468 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:46.488394 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerStarted","Data":"5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537"} Jul 14 21:52:46.488468 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:46.488460 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerStarted","Data":"00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879"} Jul 14 21:52:47.496098 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:47.496060 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerStarted","Data":"0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d"} Jul 14 21:52:47.496565 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:47.496134 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerStarted","Data":"f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f"} Jul 14 21:52:47.496565 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:47.496169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerStarted","Data":"2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1"} Jul 14 21:52:47.497498 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:47.497472 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" event={"ID":"07efdcff-0877-4700-83cd-441042289f78","Type":"ContainerStarted","Data":"038876a918def694f2476e4bf2195210f248ee023f64b3284ff29a0f1b5d7c8f"} Jul 14 21:52:47.497710 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:47.497693 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:47.502629 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:47.502610 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" Jul 14 21:52:47.515451 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:47.515399 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b77hq" podStartSLOduration=2.532025345 podStartE2EDuration="4.515380869s" totalImagesPullingTime="1.983355524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:47.514649265 +0000 UTC m=+102.364119339" watchObservedRunningTime="2026-07-14 21:52:47.515380869 +0000 UTC m=+102.364850909" Jul 14 21:52:48.505364 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:48.505271 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerStarted","Data":"59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec"} Jul 14 21:52:48.536502 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:48.536446 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.073914968 podStartE2EDuration="9.536428248s" totalImagesPullingTime="6.450465789s" totalInitContainerRuntime="12.047491ms" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:48.535743286 +0000 UTC m=+103.385213386" watchObservedRunningTime="2026-07-14 21:52:48.536428248 +0000 UTC m=+103.385898286" Jul 14 21:52:48.892776 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:48.892740 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-684f8fd54b-l6wt5"] Jul 14 21:52:48.928267 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:48.928230 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-88df566fb-ld76v"] Jul 14 21:52:48.945722 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:48.945686 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88df566fb-ld76v"] Jul 14 21:52:48.945882 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:48.945812 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.107605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.107564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-trusted-ca-bundle\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.107605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.107610 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-oauth-config\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.107827 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.107642 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-config\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.107827 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.107718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm7md\" (UniqueName: \"kubernetes.io/projected/3deefea7-ba2d-47be-bb21-91c8e31e4da8-kube-api-access-sm7md\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.107827 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.107767 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-service-ca\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.107827 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.107795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-oauth-serving-cert\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.107827 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.107825 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-serving-cert\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.208304 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.208200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-config\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.208304 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.208256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7md\" (UniqueName: \"kubernetes.io/projected/3deefea7-ba2d-47be-bb21-91c8e31e4da8-kube-api-access-sm7md\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.208538 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.208483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-service-ca\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.208538 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.208533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-oauth-serving-cert\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.208654 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.208593 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-serving-cert\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.208654 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.208636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-trusted-ca-bundle\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.208836 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.208812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-oauth-config\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.209116 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.209092 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-config\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.209299 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.209278 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-service-ca\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.209382 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.209278 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-oauth-serving-cert\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.209675 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.209655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-trusted-ca-bundle\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.211295 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.211270 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-oauth-config\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.211681 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.211634 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-serving-cert\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.217866 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.217831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7md\" (UniqueName: \"kubernetes.io/projected/3deefea7-ba2d-47be-bb21-91c8e31e4da8-kube-api-access-sm7md\") pod \"console-88df566fb-ld76v\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.255774 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.255735 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:49.422208 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.422182 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88df566fb-ld76v"] Jul 14 21:52:49.424717 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:52:49.424685 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3deefea7_ba2d_47be_bb21_91c8e31e4da8.slice/crio-c504aa677f71ee18d298d98a3ba7f1b07afd83548307b18effcae14e89833109 WatchSource:0}: Error finding container c504aa677f71ee18d298d98a3ba7f1b07afd83548307b18effcae14e89833109: Status 404 returned error can't find the container with id c504aa677f71ee18d298d98a3ba7f1b07afd83548307b18effcae14e89833109 Jul 14 21:52:49.510496 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:49.510462 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88df566fb-ld76v" event={"ID":"3deefea7-ba2d-47be-bb21-91c8e31e4da8","Type":"ContainerStarted","Data":"c504aa677f71ee18d298d98a3ba7f1b07afd83548307b18effcae14e89833109"} Jul 14 21:52:50.516058 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:50.516005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88df566fb-ld76v" event={"ID":"3deefea7-ba2d-47be-bb21-91c8e31e4da8","Type":"ContainerStarted","Data":"fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c"} Jul 14 21:52:50.538738 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:50.538682 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-88df566fb-ld76v" podStartSLOduration=2.538662136 podStartE2EDuration="2.538662136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:52:50.536399652 +0000 UTC m=+105.385869692" watchObservedRunningTime="2026-07-14 21:52:50.538662136 +0000 UTC m=+105.388132174" Jul 14 21:52:53.684673 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:53.684630 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:52:55.352982 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:55.352950 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6flwq" Jul 14 21:52:59.256043 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:59.255991 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:59.256650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:59.256079 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:59.260869 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:59.260845 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:59.553547 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:59.553518 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:52:59.614156 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:52:59.614124 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f44f987c-dmdmh"] Jul 14 21:53:00.491987 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.491919 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d5996c69c-zbrhd" podUID="4048182e-e81f-4db6-a0f6-2bc42c1e331d" containerName="console" containerID="cri-o://74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a" gracePeriod=15 Jul 14 21:53:00.770269 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.770233 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:53:00.802587 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.802556 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-service-ca\") pod \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " Jul 14 21:53:00.802587 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.802590 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-config\") pod \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " Jul 14 21:53:00.802808 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.802621 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-serving-cert\") pod \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " Jul 14 21:53:00.802808 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.802648 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-oauth-config\") pod \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " Jul 14 21:53:00.802808 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.802670 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-oauth-serving-cert\") pod \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " Jul 14 21:53:00.802808 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.802716 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hrq7\" (UniqueName: \"kubernetes.io/projected/4048182e-e81f-4db6-a0f6-2bc42c1e331d-kube-api-access-2hrq7\") pod \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\" (UID: \"4048182e-e81f-4db6-a0f6-2bc42c1e331d\") " Jul 14 21:53:00.803152 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.803120 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-config" (OuterVolumeSpecName: "console-config") pod "4048182e-e81f-4db6-a0f6-2bc42c1e331d" (UID: "4048182e-e81f-4db6-a0f6-2bc42c1e331d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:00.803152 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.803136 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4048182e-e81f-4db6-a0f6-2bc42c1e331d" (UID: "4048182e-e81f-4db6-a0f6-2bc42c1e331d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:00.803287 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.803160 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-service-ca" (OuterVolumeSpecName: "service-ca") pod "4048182e-e81f-4db6-a0f6-2bc42c1e331d" (UID: "4048182e-e81f-4db6-a0f6-2bc42c1e331d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:00.804999 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.804972 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4048182e-e81f-4db6-a0f6-2bc42c1e331d" (UID: "4048182e-e81f-4db6-a0f6-2bc42c1e331d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:53:00.805238 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.805219 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4048182e-e81f-4db6-a0f6-2bc42c1e331d-kube-api-access-2hrq7" (OuterVolumeSpecName: "kube-api-access-2hrq7") pod "4048182e-e81f-4db6-a0f6-2bc42c1e331d" (UID: "4048182e-e81f-4db6-a0f6-2bc42c1e331d"). InnerVolumeSpecName "kube-api-access-2hrq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:53:00.805308 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.805222 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4048182e-e81f-4db6-a0f6-2bc42c1e331d" (UID: "4048182e-e81f-4db6-a0f6-2bc42c1e331d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:53:00.903440 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.903400 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-serving-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:00.903440 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.903431 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-oauth-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:00.903440 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.903440 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-oauth-serving-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:00.903440 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.903449 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrq7\" (UniqueName: \"kubernetes.io/projected/4048182e-e81f-4db6-a0f6-2bc42c1e331d-kube-api-access-2hrq7\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:00.903721 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.903460 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:00.903721 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:00.903470 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4048182e-e81f-4db6-a0f6-2bc42c1e331d-console-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:01.557988 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:01.557949 2562 generic.go:356] "Generic (PLEG): container finished" podID="4048182e-e81f-4db6-a0f6-2bc42c1e331d" containerID="74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a" exitCode=0 Jul 14 21:53:01.557988 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:01.557992 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5996c69c-zbrhd" Jul 14 21:53:01.558530 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:01.558061 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5996c69c-zbrhd" event={"ID":"4048182e-e81f-4db6-a0f6-2bc42c1e331d","Type":"ContainerDied","Data":"74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a"} Jul 14 21:53:01.558530 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:01.558118 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5996c69c-zbrhd" event={"ID":"4048182e-e81f-4db6-a0f6-2bc42c1e331d","Type":"ContainerDied","Data":"12616457745debb19b3a955a370f85e138b9e5f2328ed5aa29578e67628d80e2"} Jul 14 21:53:01.558530 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:01.558140 2562 scope.go:117] "RemoveContainer" containerID="74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a" Jul 14 21:53:01.566644 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:01.566624 2562 scope.go:117] "RemoveContainer" containerID="74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a" Jul 14 21:53:01.566975 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:53:01.566952 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a\": container with ID starting with 74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a not found: ID does not exist" containerID="74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a" Jul 14 21:53:01.567037 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:01.566986 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a"} err="failed to get container status \"74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a\": rpc error: code = NotFound desc = could not find container \"74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a\": container with ID starting with 74f667a5fb4bae0c60f38ea9e61b7c87fcfbf110701b0b93e9bcb47c32f0214a not found: ID does not exist" Jul 14 21:53:01.574944 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:01.574911 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d5996c69c-zbrhd"] Jul 14 21:53:01.580327 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:01.580301 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d5996c69c-zbrhd"] Jul 14 21:53:02.809207 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:02.809172 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4048182e-e81f-4db6-a0f6-2bc42c1e331d" path="/var/lib/kubelet/pods/4048182e-e81f-4db6-a0f6-2bc42c1e331d/volumes" Jul 14 21:53:13.921864 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:13.921798 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-684f8fd54b-l6wt5" podUID="df4bd63a-9145-4a72-bace-902d06ddcae5" containerName="console" containerID="cri-o://07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7" gracePeriod=15 Jul 14 21:53:14.196827 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.196760 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:53:14.328433 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.328401 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-trusted-ca-bundle\") pod \"df4bd63a-9145-4a72-bace-902d06ddcae5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " Jul 14 21:53:14.328625 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.328447 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-console-config\") pod \"df4bd63a-9145-4a72-bace-902d06ddcae5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " Jul 14 21:53:14.328625 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.328514 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn85c\" (UniqueName: \"kubernetes.io/projected/df4bd63a-9145-4a72-bace-902d06ddcae5-kube-api-access-hn85c\") pod \"df4bd63a-9145-4a72-bace-902d06ddcae5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " Jul 14 21:53:14.328625 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.328563 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-oauth-config\") pod \"df4bd63a-9145-4a72-bace-902d06ddcae5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " Jul 14 21:53:14.328625 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.328593 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-service-ca\") pod \"df4bd63a-9145-4a72-bace-902d06ddcae5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " Jul 14 21:53:14.328625 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.328624 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-serving-cert\") pod \"df4bd63a-9145-4a72-bace-902d06ddcae5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " Jul 14 21:53:14.328861 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.328657 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-oauth-serving-cert\") pod \"df4bd63a-9145-4a72-bace-902d06ddcae5\" (UID: \"df4bd63a-9145-4a72-bace-902d06ddcae5\") " Jul 14 21:53:14.328861 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.328811 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "df4bd63a-9145-4a72-bace-902d06ddcae5" (UID: "df4bd63a-9145-4a72-bace-902d06ddcae5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:14.329051 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.328869 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-console-config" (OuterVolumeSpecName: "console-config") pod "df4bd63a-9145-4a72-bace-902d06ddcae5" (UID: "df4bd63a-9145-4a72-bace-902d06ddcae5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:14.329051 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.328918 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-trusted-ca-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:14.329158 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.329060 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "df4bd63a-9145-4a72-bace-902d06ddcae5" (UID: "df4bd63a-9145-4a72-bace-902d06ddcae5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:14.329328 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.329258 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-service-ca" (OuterVolumeSpecName: "service-ca") pod "df4bd63a-9145-4a72-bace-902d06ddcae5" (UID: "df4bd63a-9145-4a72-bace-902d06ddcae5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:14.330873 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.330847 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "df4bd63a-9145-4a72-bace-902d06ddcae5" (UID: "df4bd63a-9145-4a72-bace-902d06ddcae5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:53:14.330873 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.330865 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "df4bd63a-9145-4a72-bace-902d06ddcae5" (UID: "df4bd63a-9145-4a72-bace-902d06ddcae5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:53:14.331207 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.331188 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4bd63a-9145-4a72-bace-902d06ddcae5-kube-api-access-hn85c" (OuterVolumeSpecName: "kube-api-access-hn85c") pod "df4bd63a-9145-4a72-bace-902d06ddcae5" (UID: "df4bd63a-9145-4a72-bace-902d06ddcae5"). InnerVolumeSpecName "kube-api-access-hn85c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:53:14.429657 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.429617 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:14.429657 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.429650 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-serving-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:14.429657 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.429662 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-oauth-serving-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:14.429891 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.429671 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df4bd63a-9145-4a72-bace-902d06ddcae5-console-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:14.429891 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.429679 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn85c\" (UniqueName: \"kubernetes.io/projected/df4bd63a-9145-4a72-bace-902d06ddcae5-kube-api-access-hn85c\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:14.429891 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.429687 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df4bd63a-9145-4a72-bace-902d06ddcae5-console-oauth-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:14.610596 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.610562 2562 generic.go:356] "Generic (PLEG): container finished" podID="df4bd63a-9145-4a72-bace-902d06ddcae5" containerID="07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7" exitCode=0 Jul 14 21:53:14.610782 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.610617 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684f8fd54b-l6wt5" Jul 14 21:53:14.610782 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.610627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684f8fd54b-l6wt5" event={"ID":"df4bd63a-9145-4a72-bace-902d06ddcae5","Type":"ContainerDied","Data":"07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7"} Jul 14 21:53:14.610782 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.610674 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684f8fd54b-l6wt5" event={"ID":"df4bd63a-9145-4a72-bace-902d06ddcae5","Type":"ContainerDied","Data":"3aed4e327026ab4c77ec3e55d2c2c3518b15ffc34b6c1540fe7d94824de3547f"} Jul 14 21:53:14.610782 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.610697 2562 scope.go:117] "RemoveContainer" containerID="07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7" Jul 14 21:53:14.618899 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.618883 2562 scope.go:117] "RemoveContainer" containerID="07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7" Jul 14 21:53:14.619185 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:53:14.619164 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7\": container with ID starting with 07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7 not found: ID does not exist" containerID="07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7" Jul 14 21:53:14.619240 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.619195 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7"} err="failed to get container status \"07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7\": rpc error: code = NotFound desc = could not find container \"07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7\": container with ID starting with 07c51122a7eb8fb1371932515743aff5a441a7fd32332c9f10c1e30e217bf0f7 not found: ID does not exist" Jul 14 21:53:14.632302 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.632280 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-684f8fd54b-l6wt5"] Jul 14 21:53:14.637774 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.637753 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-684f8fd54b-l6wt5"] Jul 14 21:53:14.809246 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:14.809206 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df4bd63a-9145-4a72-bace-902d06ddcae5" path="/var/lib/kubelet/pods/df4bd63a-9145-4a72-bace-902d06ddcae5/volumes" Jul 14 21:53:24.634593 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:24.634530 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86f44f987c-dmdmh" podUID="33a1ec5b-07ec-4853-bb95-1bf77802e511" containerName="console" containerID="cri-o://61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8" gracePeriod=15 Jul 14 21:53:24.927768 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:24.927742 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:53:25.022655 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.022621 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-oauth-config\") pod \"33a1ec5b-07ec-4853-bb95-1bf77802e511\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " Jul 14 21:53:25.022865 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.022680 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-serving-cert\") pod \"33a1ec5b-07ec-4853-bb95-1bf77802e511\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " Jul 14 21:53:25.022865 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.022713 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-oauth-serving-cert\") pod \"33a1ec5b-07ec-4853-bb95-1bf77802e511\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " Jul 14 21:53:25.022865 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.022749 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-service-ca\") pod \"33a1ec5b-07ec-4853-bb95-1bf77802e511\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " Jul 14 21:53:25.022865 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.022777 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-config\") pod \"33a1ec5b-07ec-4853-bb95-1bf77802e511\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " Jul 14 21:53:25.023116 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.022946 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl4d4\" (UniqueName: \"kubernetes.io/projected/33a1ec5b-07ec-4853-bb95-1bf77802e511-kube-api-access-pl4d4\") pod \"33a1ec5b-07ec-4853-bb95-1bf77802e511\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " Jul 14 21:53:25.023116 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.023007 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-trusted-ca-bundle\") pod \"33a1ec5b-07ec-4853-bb95-1bf77802e511\" (UID: \"33a1ec5b-07ec-4853-bb95-1bf77802e511\") " Jul 14 21:53:25.023226 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.023157 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-service-ca" (OuterVolumeSpecName: "service-ca") pod "33a1ec5b-07ec-4853-bb95-1bf77802e511" (UID: "33a1ec5b-07ec-4853-bb95-1bf77802e511"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:25.023226 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.023177 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "33a1ec5b-07ec-4853-bb95-1bf77802e511" (UID: "33a1ec5b-07ec-4853-bb95-1bf77802e511"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:25.023226 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.023211 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-config" (OuterVolumeSpecName: "console-config") pod "33a1ec5b-07ec-4853-bb95-1bf77802e511" (UID: "33a1ec5b-07ec-4853-bb95-1bf77802e511"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:25.023444 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.023421 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:25.023511 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.023447 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:25.023511 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.023460 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-oauth-serving-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:25.023593 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.023506 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "33a1ec5b-07ec-4853-bb95-1bf77802e511" (UID: "33a1ec5b-07ec-4853-bb95-1bf77802e511"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:53:25.024927 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.024896 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "33a1ec5b-07ec-4853-bb95-1bf77802e511" (UID: "33a1ec5b-07ec-4853-bb95-1bf77802e511"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:53:25.025094 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.024937 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "33a1ec5b-07ec-4853-bb95-1bf77802e511" (UID: "33a1ec5b-07ec-4853-bb95-1bf77802e511"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:53:25.025094 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.024958 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a1ec5b-07ec-4853-bb95-1bf77802e511-kube-api-access-pl4d4" (OuterVolumeSpecName: "kube-api-access-pl4d4") pod "33a1ec5b-07ec-4853-bb95-1bf77802e511" (UID: "33a1ec5b-07ec-4853-bb95-1bf77802e511"). InnerVolumeSpecName "kube-api-access-pl4d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:53:25.124079 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.124040 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl4d4\" (UniqueName: \"kubernetes.io/projected/33a1ec5b-07ec-4853-bb95-1bf77802e511-kube-api-access-pl4d4\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:25.124079 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.124079 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33a1ec5b-07ec-4853-bb95-1bf77802e511-trusted-ca-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:25.124317 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.124095 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-oauth-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:25.124317 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.124109 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33a1ec5b-07ec-4853-bb95-1bf77802e511-console-serving-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:53:25.654576 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.654541 2562 generic.go:356] "Generic (PLEG): container finished" podID="33a1ec5b-07ec-4853-bb95-1bf77802e511" containerID="61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8" exitCode=0 Jul 14 21:53:25.654989 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.654604 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f44f987c-dmdmh" Jul 14 21:53:25.654989 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.654620 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f44f987c-dmdmh" event={"ID":"33a1ec5b-07ec-4853-bb95-1bf77802e511","Type":"ContainerDied","Data":"61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8"} Jul 14 21:53:25.654989 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.654685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f44f987c-dmdmh" event={"ID":"33a1ec5b-07ec-4853-bb95-1bf77802e511","Type":"ContainerDied","Data":"0033addc5989d67d89a52f22c0335a9ff19c36474bc753bf8bc5c9b3afd03304"} Jul 14 21:53:25.654989 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.654704 2562 scope.go:117] "RemoveContainer" containerID="61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8" Jul 14 21:53:25.663407 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.663390 2562 scope.go:117] "RemoveContainer" containerID="61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8" Jul 14 21:53:25.663661 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:53:25.663644 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8\": container with ID starting with 61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8 not found: ID does not exist" containerID="61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8" Jul 14 21:53:25.663712 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.663668 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8"} err="failed to get container status \"61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8\": rpc error: code = NotFound desc = could not find container \"61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8\": container with ID starting with 61b430b05f7d2bbb26e8e18e952595d897d7cab97a6c676daba9c1aaafed9ca8 not found: ID does not exist" Jul 14 21:53:25.677070 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.677045 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f44f987c-dmdmh"] Jul 14 21:53:25.681660 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:25.681639 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86f44f987c-dmdmh"] Jul 14 21:53:26.808914 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:26.808882 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a1ec5b-07ec-4853-bb95-1bf77802e511" path="/var/lib/kubelet/pods/33a1ec5b-07ec-4853-bb95-1bf77802e511/volumes" Jul 14 21:53:58.971706 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:58.971622 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 21:53:58.972222 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:58.971941 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="alertmanager" containerID="cri-o://00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879" gracePeriod=120 Jul 14 21:53:58.972222 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:58.971998 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy-metric" containerID="cri-o://0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d" gracePeriod=120 Jul 14 21:53:58.972222 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:58.972051 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="prom-label-proxy" containerID="cri-o://59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec" gracePeriod=120 Jul 14 21:53:58.972222 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:58.972096 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="config-reloader" containerID="cri-o://5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537" gracePeriod=120 Jul 14 21:53:58.972222 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:58.972132 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy" containerID="cri-o://f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f" gracePeriod=120 Jul 14 21:53:58.972222 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:58.972076 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy-web" containerID="cri-o://2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1" gracePeriod=120 Jul 14 21:53:59.787860 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:59.787828 2562 generic.go:356] "Generic (PLEG): container finished" podID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerID="59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec" exitCode=0 Jul 14 21:53:59.787860 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:59.787850 2562 generic.go:356] "Generic (PLEG): container finished" podID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerID="f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f" exitCode=0 Jul 14 21:53:59.787860 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:59.787856 2562 generic.go:356] "Generic (PLEG): container finished" podID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerID="5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537" exitCode=0 Jul 14 21:53:59.787860 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:59.787862 2562 generic.go:356] "Generic (PLEG): container finished" podID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerID="00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879" exitCode=0 Jul 14 21:53:59.788151 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:59.787896 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerDied","Data":"59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec"} Jul 14 21:53:59.788151 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:59.787959 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerDied","Data":"f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f"} Jul 14 21:53:59.788151 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:59.787987 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerDied","Data":"5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537"} Jul 14 21:53:59.788151 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:53:59.788011 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerDied","Data":"00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879"} Jul 14 21:54:00.215461 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.215438 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.293293 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293263 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293480 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293299 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-config-volume\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293480 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293332 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-cluster-tls-config\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293480 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293362 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-main-tls\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293480 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293386 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksshl\" (UniqueName: \"kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-kube-api-access-ksshl\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293480 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293419 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293717 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293605 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-web-config\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293717 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293655 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-tls-assets\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293717 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293695 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-trusted-ca-bundle\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293866 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293727 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-metrics-client-ca\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293866 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293754 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-web\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.293866 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293792 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-main-db\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.294005 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.293866 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-config-out\") pod \"6ccd7939-3a63-458c-b21c-9236c7228b29\" (UID: \"6ccd7939-3a63-458c-b21c-9236c7228b29\") " Jul 14 21:54:00.294512 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.294483 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:54:00.295129 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.295093 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:54:00.295540 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.295515 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:54:00.296851 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.296821 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:54:00.298038 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.298001 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-config-volume" (OuterVolumeSpecName: "config-volume") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:54:00.298403 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.298373 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:54:00.298512 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.298406 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:54:00.298512 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.298466 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:54:00.298512 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.298471 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-config-out" (OuterVolumeSpecName: "config-out") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:54:00.298911 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.298885 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:54:00.299740 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.299716 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-kube-api-access-ksshl" (OuterVolumeSpecName: "kube-api-access-ksshl") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "kube-api-access-ksshl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:54:00.301605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.301580 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:54:00.309306 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.309282 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-web-config" (OuterVolumeSpecName: "web-config") pod "6ccd7939-3a63-458c-b21c-9236c7228b29" (UID: "6ccd7939-3a63-458c-b21c-9236c7228b29"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:54:00.376655 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376570 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6679c7c685-zmvmj"] Jul 14 21:54:00.376885 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376872 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="prom-label-proxy" Jul 14 21:54:00.376924 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376886 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="prom-label-proxy" Jul 14 21:54:00.376924 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376896 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4048182e-e81f-4db6-a0f6-2bc42c1e331d" containerName="console" Jul 14 21:54:00.376924 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376902 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4048182e-e81f-4db6-a0f6-2bc42c1e331d" containerName="console" Jul 14 21:54:00.376924 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376911 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="config-reloader" Jul 14 21:54:00.376924 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376917 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="config-reloader" Jul 14 21:54:00.376924 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376924 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy-web" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376930 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy-web" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376941 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="init-config-reloader" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376946 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="init-config-reloader" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376951 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df4bd63a-9145-4a72-bace-902d06ddcae5" containerName="console" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376956 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="df4bd63a-9145-4a72-bace-902d06ddcae5" containerName="console" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376967 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376973 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376978 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy-metric" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376984 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy-metric" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376992 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="alertmanager" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.376998 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="alertmanager" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377004 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a1ec5b-07ec-4853-bb95-1bf77802e511" containerName="console" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377009 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a1ec5b-07ec-4853-bb95-1bf77802e511" containerName="console" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377069 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="config-reloader" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377077 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377083 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="alertmanager" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377089 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4048182e-e81f-4db6-a0f6-2bc42c1e331d" containerName="console" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377094 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy-metric" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377099 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="prom-label-proxy" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377105 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a1ec5b-07ec-4853-bb95-1bf77802e511" containerName="console" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377110 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="df4bd63a-9145-4a72-bace-902d06ddcae5" containerName="console" Jul 14 21:54:00.377130 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.377114 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerName="kube-rbac-proxy-web" Jul 14 21:54:00.379326 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.379309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.393574 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.393547 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6679c7c685-zmvmj"] Jul 14 21:54:00.395299 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssm6n\" (UniqueName: \"kubernetes.io/projected/85a80b20-95a0-455c-b391-5dab2112d6cd-kube-api-access-ssm6n\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.395411 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-service-ca\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.395411 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-serving-cert\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.395411 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395381 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-trusted-ca-bundle\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.395411 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-oauth-config\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.395569 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-console-config\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.395569 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395469 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-oauth-serving-cert\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.395569 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395517 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395569 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395528 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ccd7939-3a63-458c-b21c-9236c7228b29-metrics-client-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395569 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395539 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395569 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395549 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-alertmanager-main-db\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395569 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395565 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ccd7939-3a63-458c-b21c-9236c7228b29-config-out\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395576 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395584 2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-config-volume\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395592 2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-cluster-tls-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395601 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-main-tls\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395609 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksshl\" (UniqueName: \"kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-kube-api-access-ksshl\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395618 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395626 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ccd7939-3a63-458c-b21c-9236c7228b29-web-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.395772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.395634 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ccd7939-3a63-458c-b21c-9236c7228b29-tls-assets\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:00.496733 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.496690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-trusted-ca-bundle\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.496733 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.496732 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-oauth-config\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.496987 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.496755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-console-config\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.496987 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.496797 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-oauth-serving-cert\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.496987 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.496827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssm6n\" (UniqueName: \"kubernetes.io/projected/85a80b20-95a0-455c-b391-5dab2112d6cd-kube-api-access-ssm6n\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.496987 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.496852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-service-ca\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.496987 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.496884 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-serving-cert\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.497629 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.497597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-service-ca\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.497757 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.497734 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-console-config\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.497818 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.497734 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-trusted-ca-bundle\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.497818 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.497801 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-oauth-serving-cert\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.499271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.499249 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-oauth-config\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.499384 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.499367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-serving-cert\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.509616 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.509587 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssm6n\" (UniqueName: \"kubernetes.io/projected/85a80b20-95a0-455c-b391-5dab2112d6cd-kube-api-access-ssm6n\") pod \"console-6679c7c685-zmvmj\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.688583 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.688485 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:00.796347 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.796312 2562 generic.go:356] "Generic (PLEG): container finished" podID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerID="0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d" exitCode=0 Jul 14 21:54:00.796347 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.796340 2562 generic.go:356] "Generic (PLEG): container finished" podID="6ccd7939-3a63-458c-b21c-9236c7228b29" containerID="2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1" exitCode=0 Jul 14 21:54:00.796566 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.796365 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerDied","Data":"0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d"} Jul 14 21:54:00.796566 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.796446 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.796566 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.796460 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerDied","Data":"2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1"} Jul 14 21:54:00.796566 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.796491 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ccd7939-3a63-458c-b21c-9236c7228b29","Type":"ContainerDied","Data":"f5f87361e2c82a9da48edeea962b9bd87abc7296e2d594187add31786deb3304"} Jul 14 21:54:00.796566 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.796524 2562 scope.go:117] "RemoveContainer" containerID="59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec" Jul 14 21:54:00.804727 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.804703 2562 scope.go:117] "RemoveContainer" containerID="0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d" Jul 14 21:54:00.812548 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.812531 2562 scope.go:117] "RemoveContainer" containerID="f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f" Jul 14 21:54:00.819202 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.819184 2562 scope.go:117] "RemoveContainer" containerID="2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1" Jul 14 21:54:00.821960 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.821939 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 21:54:00.826383 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.826356 2562 scope.go:117] "RemoveContainer" containerID="5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537" Jul 14 21:54:00.830899 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.830875 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6679c7c685-zmvmj"] Jul 14 21:54:00.832424 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.832400 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 21:54:00.835672 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.835654 2562 scope.go:117] "RemoveContainer" containerID="00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879" Jul 14 21:54:00.836712 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:54:00.836650 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a80b20_95a0_455c_b391_5dab2112d6cd.slice/crio-c63905c2beb0cd115a3f723d9a5898b02cf9b6cb7a1319ee61066fbbc3b05b6f WatchSource:0}: Error finding container c63905c2beb0cd115a3f723d9a5898b02cf9b6cb7a1319ee61066fbbc3b05b6f: Status 404 returned error can't find the container with id c63905c2beb0cd115a3f723d9a5898b02cf9b6cb7a1319ee61066fbbc3b05b6f Jul 14 21:54:00.843309 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.843286 2562 scope.go:117] "RemoveContainer" containerID="0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930" Jul 14 21:54:00.850549 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.850524 2562 scope.go:117] "RemoveContainer" containerID="59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec" Jul 14 21:54:00.850893 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:54:00.850871 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec\": container with ID starting with 59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec not found: ID does not exist" containerID="59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec" Jul 14 21:54:00.850941 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.850907 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec"} err="failed to get container status \"59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec\": rpc error: code = NotFound desc = could not find container \"59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec\": container with ID starting with 59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec not found: ID does not exist" Jul 14 21:54:00.850941 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.850935 2562 scope.go:117] "RemoveContainer" containerID="0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d" Jul 14 21:54:00.851245 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:54:00.851227 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d\": container with ID starting with 0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d not found: ID does not exist" containerID="0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d" Jul 14 21:54:00.851307 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.851252 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d"} err="failed to get container status \"0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d\": rpc error: code = NotFound desc = could not find container \"0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d\": container with ID starting with 0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d not found: ID does not exist" Jul 14 21:54:00.851307 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.851270 2562 scope.go:117] "RemoveContainer" containerID="f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f" Jul 14 21:54:00.851499 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:54:00.851480 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f\": container with ID starting with f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f not found: ID does not exist" containerID="f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f" Jul 14 21:54:00.851567 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.851505 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f"} err="failed to get container status \"f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f\": rpc error: code = NotFound desc = could not find container \"f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f\": container with ID starting with f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f not found: ID does not exist" Jul 14 21:54:00.851567 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.851523 2562 scope.go:117] "RemoveContainer" containerID="2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1" Jul 14 21:54:00.851775 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:54:00.851760 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1\": container with ID starting with 2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1 not found: ID does not exist" containerID="2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1" Jul 14 21:54:00.851812 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.851779 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1"} err="failed to get container status \"2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1\": rpc error: code = NotFound desc = could not find container \"2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1\": container with ID starting with 2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1 not found: ID does not exist" Jul 14 21:54:00.851812 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.851795 2562 scope.go:117] "RemoveContainer" containerID="5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537" Jul 14 21:54:00.852016 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:54:00.851998 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537\": container with ID starting with 5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537 not found: ID does not exist" containerID="5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537" Jul 14 21:54:00.852103 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.852038 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537"} err="failed to get container status \"5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537\": rpc error: code = NotFound desc = could not find container \"5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537\": container with ID starting with 5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537 not found: ID does not exist" Jul 14 21:54:00.852103 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.852061 2562 scope.go:117] "RemoveContainer" containerID="00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879" Jul 14 21:54:00.852329 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:54:00.852313 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879\": container with ID starting with 00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879 not found: ID does not exist" containerID="00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879" Jul 14 21:54:00.852378 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.852333 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879"} err="failed to get container status \"00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879\": rpc error: code = NotFound desc = could not find container \"00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879\": container with ID starting with 00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879 not found: ID does not exist" Jul 14 21:54:00.852378 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.852346 2562 scope.go:117] "RemoveContainer" containerID="0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930" Jul 14 21:54:00.852642 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:54:00.852621 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930\": container with ID starting with 0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930 not found: ID does not exist" containerID="0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930" Jul 14 21:54:00.852689 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.852649 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930"} err="failed to get container status \"0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930\": rpc error: code = NotFound desc = could not find container \"0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930\": container with ID starting with 0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930 not found: ID does not exist" Jul 14 21:54:00.852689 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.852669 2562 scope.go:117] "RemoveContainer" containerID="59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec" Jul 14 21:54:00.852923 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.852905 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec"} err="failed to get container status \"59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec\": rpc error: code = NotFound desc = could not find container \"59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec\": container with ID starting with 59ad3938e6c4849f1330092ebb602be32864034bb314b39ef9f9b79faed325ec not found: ID does not exist" Jul 14 21:54:00.852961 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.852925 2562 scope.go:117] "RemoveContainer" containerID="0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d" Jul 14 21:54:00.853163 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.853143 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d"} err="failed to get container status \"0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d\": rpc error: code = NotFound desc = could not find container \"0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d\": container with ID starting with 0bee149f4b6c3abbde1ea94f2b2b4f2f7d20c36f44de009012df823fbb86d80d not found: ID does not exist" Jul 14 21:54:00.853223 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.853164 2562 scope.go:117] "RemoveContainer" containerID="f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f" Jul 14 21:54:00.853392 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.853377 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f"} err="failed to get container status \"f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f\": rpc error: code = NotFound desc = could not find container \"f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f\": container with ID starting with f601e5636d220866c2416f13eec4c11ba359e8fd7b3be6299140002f2f721e3f not found: ID does not exist" Jul 14 21:54:00.853441 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.853392 2562 scope.go:117] "RemoveContainer" containerID="2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1" Jul 14 21:54:00.853620 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.853589 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1"} err="failed to get container status \"2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1\": rpc error: code = NotFound desc = could not find container \"2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1\": container with ID starting with 2f41c8c5a92bdd498182b99727cdf921eed99cae0f43740158d015c9c540d4b1 not found: ID does not exist" Jul 14 21:54:00.853620 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.853619 2562 scope.go:117] "RemoveContainer" containerID="5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537" Jul 14 21:54:00.853859 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.853838 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537"} err="failed to get container status \"5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537\": rpc error: code = NotFound desc = could not find container \"5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537\": container with ID starting with 5ad98eba214a87d54688f8c6868772eb69f16c7eb9b288436e68f624d4a31537 not found: ID does not exist" Jul 14 21:54:00.853933 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.853860 2562 scope.go:117] "RemoveContainer" containerID="00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879" Jul 14 21:54:00.854139 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.854117 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879"} err="failed to get container status \"00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879\": rpc error: code = NotFound desc = could not find container \"00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879\": container with ID starting with 00f159f491f95a7bfa4bd00b5a10c50bdaed9db9379fdf4ee990b8062e2c6879 not found: ID does not exist" Jul 14 21:54:00.854237 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.854139 2562 scope.go:117] "RemoveContainer" containerID="0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930" Jul 14 21:54:00.854373 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.854355 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930"} err="failed to get container status \"0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930\": rpc error: code = NotFound desc = could not find container \"0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930\": container with ID starting with 0da579332436b8b40907e908e598398f6e8a47f901db334a83f911a9fc7f5930 not found: ID does not exist" Jul 14 21:54:00.861226 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.861206 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 21:54:00.864748 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.864731 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.868776 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.868755 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 14 21:54:00.869327 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.868756 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 14 21:54:00.869327 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.868766 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 14 21:54:00.869327 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.869004 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 14 21:54:00.869327 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.869006 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n5q5n\"" Jul 14 21:54:00.869327 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.869164 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 14 21:54:00.869803 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.869393 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 14 21:54:00.869803 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.869596 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 14 21:54:00.869998 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.869971 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 14 21:54:00.873910 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.873892 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 14 21:54:00.881415 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.881396 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 21:54:00.900822 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.900800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.900926 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.900837 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-web-config\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.900926 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.900870 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.901056 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.900945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.901056 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.900974 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b405286f-706e-4f3b-8484-5dfb9284d912-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.901056 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.901004 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b405286f-706e-4f3b-8484-5dfb9284d912-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.901056 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.901039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-config-volume\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.901271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.901064 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b405286f-706e-4f3b-8484-5dfb9284d912-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.901271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.901099 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b405286f-706e-4f3b-8484-5dfb9284d912-config-out\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.901271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.901123 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.901271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.901147 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.901271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.901168 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b405286f-706e-4f3b-8484-5dfb9284d912-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:00.901271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:00.901194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57vf\" (UniqueName: \"kubernetes.io/projected/b405286f-706e-4f3b-8484-5dfb9284d912-kube-api-access-p57vf\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.002768 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.002768 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-web-config\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.002768 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.002768 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.003122 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002788 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b405286f-706e-4f3b-8484-5dfb9284d912-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.003122 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b405286f-706e-4f3b-8484-5dfb9284d912-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.003122 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-config-volume\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.003122 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002919 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b405286f-706e-4f3b-8484-5dfb9284d912-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.003122 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b405286f-706e-4f3b-8484-5dfb9284d912-config-out\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.003122 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002972 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.003122 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.002992 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.003122 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.003014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b405286f-706e-4f3b-8484-5dfb9284d912-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.003122 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.003051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57vf\" (UniqueName: \"kubernetes.io/projected/b405286f-706e-4f3b-8484-5dfb9284d912-kube-api-access-p57vf\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.003577 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.003536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b405286f-706e-4f3b-8484-5dfb9284d912-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.004649 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.004621 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b405286f-706e-4f3b-8484-5dfb9284d912-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.004969 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.004941 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b405286f-706e-4f3b-8484-5dfb9284d912-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.006310 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.005989 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b405286f-706e-4f3b-8484-5dfb9284d912-config-out\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.006586 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.006511 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.006586 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.006567 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.006737 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.006694 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-web-config\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.006737 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.006708 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.006834 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.006771 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.007281 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.007256 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-config-volume\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.007672 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.007654 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b405286f-706e-4f3b-8484-5dfb9284d912-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.007900 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.007886 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b405286f-706e-4f3b-8484-5dfb9284d912-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.011592 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.011570 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57vf\" (UniqueName: \"kubernetes.io/projected/b405286f-706e-4f3b-8484-5dfb9284d912-kube-api-access-p57vf\") pod \"alertmanager-main-0\" (UID: \"b405286f-706e-4f3b-8484-5dfb9284d912\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.191779 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.191736 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 21:54:01.346820 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.346789 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 21:54:01.349721 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:54:01.349696 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb405286f_706e_4f3b_8484_5dfb9284d912.slice/crio-4d877194d383ccf4960de923ec2b2ae23a1045ddbc7585a4c6479007ecfd8c29 WatchSource:0}: Error finding container 4d877194d383ccf4960de923ec2b2ae23a1045ddbc7585a4c6479007ecfd8c29: Status 404 returned error can't find the container with id 4d877194d383ccf4960de923ec2b2ae23a1045ddbc7585a4c6479007ecfd8c29 Jul 14 21:54:01.801893 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.801854 2562 generic.go:356] "Generic (PLEG): container finished" podID="b405286f-706e-4f3b-8484-5dfb9284d912" containerID="d87b57e70baaecd8238bbe9982bb90f05f93ac75ec5d774993501ba42d7bfdae" exitCode=0 Jul 14 21:54:01.802087 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.801926 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b405286f-706e-4f3b-8484-5dfb9284d912","Type":"ContainerDied","Data":"d87b57e70baaecd8238bbe9982bb90f05f93ac75ec5d774993501ba42d7bfdae"} Jul 14 21:54:01.802087 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.802010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b405286f-706e-4f3b-8484-5dfb9284d912","Type":"ContainerStarted","Data":"4d877194d383ccf4960de923ec2b2ae23a1045ddbc7585a4c6479007ecfd8c29"} Jul 14 21:54:01.803572 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.803544 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6679c7c685-zmvmj" event={"ID":"85a80b20-95a0-455c-b391-5dab2112d6cd","Type":"ContainerStarted","Data":"3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce"} Jul 14 21:54:01.803698 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.803585 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6679c7c685-zmvmj" event={"ID":"85a80b20-95a0-455c-b391-5dab2112d6cd","Type":"ContainerStarted","Data":"c63905c2beb0cd115a3f723d9a5898b02cf9b6cb7a1319ee61066fbbc3b05b6f"} Jul 14 21:54:01.847687 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:01.847641 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6679c7c685-zmvmj" podStartSLOduration=1.847624428 podStartE2EDuration="1.847624428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:54:01.846471431 +0000 UTC m=+176.695941499" watchObservedRunningTime="2026-07-14 21:54:01.847624428 +0000 UTC m=+176.697094470" Jul 14 21:54:02.812497 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:02.812464 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ccd7939-3a63-458c-b21c-9236c7228b29" path="/var/lib/kubelet/pods/6ccd7939-3a63-458c-b21c-9236c7228b29/volumes" Jul 14 21:54:02.814730 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:02.814700 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b405286f-706e-4f3b-8484-5dfb9284d912","Type":"ContainerStarted","Data":"4880f9c9d155187b784d0ad55a33d36b04dd40c61b5c2953a9b3b106cbe5cee6"} Jul 14 21:54:02.814866 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:02.814762 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b405286f-706e-4f3b-8484-5dfb9284d912","Type":"ContainerStarted","Data":"d3d2440de9ffdbacea5e851e6d62f00728601a46389097f186aa9387c46f90ad"} Jul 14 21:54:02.814866 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:02.814795 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b405286f-706e-4f3b-8484-5dfb9284d912","Type":"ContainerStarted","Data":"dd6ca7506660f734ab26f54ac2e571101e67e91f3a24ba712f1b3f162d2a0763"} Jul 14 21:54:02.814866 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:02.814823 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b405286f-706e-4f3b-8484-5dfb9284d912","Type":"ContainerStarted","Data":"4daa69cb035ff9e38bf416d2db010d83046216c29fbc796755b25310571ccc5a"} Jul 14 21:54:02.814866 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:02.814847 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b405286f-706e-4f3b-8484-5dfb9284d912","Type":"ContainerStarted","Data":"7f0247edb24438902a89a45af0faee9e93d13b5ddb9b53cf210a49884cd7713b"} Jul 14 21:54:02.815058 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:02.814871 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b405286f-706e-4f3b-8484-5dfb9284d912","Type":"ContainerStarted","Data":"3ff1d050b370fbf91b4d3c7c664bc73c0428542a584d7a9e4260333b9134f33d"} Jul 14 21:54:02.850415 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:02.850356 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.834150678 podStartE2EDuration="2.850338526s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.187848ms" isStatefulPod=false podCreationTimestamp="2026-07-14 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:54:02.847745035 +0000 UTC m=+177.697215130" watchObservedRunningTime="2026-07-14 21:54:02.850338526 +0000 UTC m=+177.699808564" Jul 14 21:54:10.688878 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:10.688836 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:10.689331 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:10.688907 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:10.693896 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:10.693871 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:10.851848 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:10.851819 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:54:10.912504 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:10.912472 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88df566fb-ld76v"] Jul 14 21:54:35.933475 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:35.933403 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-88df566fb-ld76v" podUID="3deefea7-ba2d-47be-bb21-91c8e31e4da8" containerName="console" containerID="cri-o://fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c" gracePeriod=15 Jul 14 21:54:36.171341 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.171316 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:54:36.308872 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.308838 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-config\") pod \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " Jul 14 21:54:36.309066 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.308890 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-serving-cert\") pod \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " Jul 14 21:54:36.309066 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.308919 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-oauth-config\") pod \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " Jul 14 21:54:36.309066 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.308958 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-oauth-serving-cert\") pod \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " Jul 14 21:54:36.309066 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.308990 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm7md\" (UniqueName: \"kubernetes.io/projected/3deefea7-ba2d-47be-bb21-91c8e31e4da8-kube-api-access-sm7md\") pod \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " Jul 14 21:54:36.309271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.309078 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-trusted-ca-bundle\") pod \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " Jul 14 21:54:36.309271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.309105 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-service-ca\") pod \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\" (UID: \"3deefea7-ba2d-47be-bb21-91c8e31e4da8\") " Jul 14 21:54:36.309558 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.309375 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-config" (OuterVolumeSpecName: "console-config") pod "3deefea7-ba2d-47be-bb21-91c8e31e4da8" (UID: "3deefea7-ba2d-47be-bb21-91c8e31e4da8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:54:36.309558 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.309520 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3deefea7-ba2d-47be-bb21-91c8e31e4da8" (UID: "3deefea7-ba2d-47be-bb21-91c8e31e4da8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:54:36.309558 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.309540 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3deefea7-ba2d-47be-bb21-91c8e31e4da8" (UID: "3deefea7-ba2d-47be-bb21-91c8e31e4da8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:54:36.309828 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.309563 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-service-ca" (OuterVolumeSpecName: "service-ca") pod "3deefea7-ba2d-47be-bb21-91c8e31e4da8" (UID: "3deefea7-ba2d-47be-bb21-91c8e31e4da8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:54:36.311642 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.311603 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3deefea7-ba2d-47be-bb21-91c8e31e4da8" (UID: "3deefea7-ba2d-47be-bb21-91c8e31e4da8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:54:36.311889 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.311851 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3deefea7-ba2d-47be-bb21-91c8e31e4da8-kube-api-access-sm7md" (OuterVolumeSpecName: "kube-api-access-sm7md") pod "3deefea7-ba2d-47be-bb21-91c8e31e4da8" (UID: "3deefea7-ba2d-47be-bb21-91c8e31e4da8"). InnerVolumeSpecName "kube-api-access-sm7md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:54:36.317087 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.317061 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3deefea7-ba2d-47be-bb21-91c8e31e4da8" (UID: "3deefea7-ba2d-47be-bb21-91c8e31e4da8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:54:36.410145 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.410104 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:36.410145 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.410138 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-serving-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:36.410145 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.410148 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3deefea7-ba2d-47be-bb21-91c8e31e4da8-console-oauth-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:36.410145 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.410158 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-oauth-serving-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:36.410410 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.410167 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm7md\" (UniqueName: \"kubernetes.io/projected/3deefea7-ba2d-47be-bb21-91c8e31e4da8-kube-api-access-sm7md\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:36.410410 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.410176 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-trusted-ca-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:36.410410 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.410184 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3deefea7-ba2d-47be-bb21-91c8e31e4da8-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:54:36.947661 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.947625 2562 generic.go:356] "Generic (PLEG): container finished" podID="3deefea7-ba2d-47be-bb21-91c8e31e4da8" containerID="fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c" exitCode=0 Jul 14 21:54:36.948162 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.947685 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88df566fb-ld76v" Jul 14 21:54:36.948162 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.947714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88df566fb-ld76v" event={"ID":"3deefea7-ba2d-47be-bb21-91c8e31e4da8","Type":"ContainerDied","Data":"fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c"} Jul 14 21:54:36.948162 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.947779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88df566fb-ld76v" event={"ID":"3deefea7-ba2d-47be-bb21-91c8e31e4da8","Type":"ContainerDied","Data":"c504aa677f71ee18d298d98a3ba7f1b07afd83548307b18effcae14e89833109"} Jul 14 21:54:36.948162 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.947810 2562 scope.go:117] "RemoveContainer" containerID="fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c" Jul 14 21:54:36.956112 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.956095 2562 scope.go:117] "RemoveContainer" containerID="fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c" Jul 14 21:54:36.956424 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:54:36.956404 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c\": container with ID starting with fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c not found: ID does not exist" containerID="fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c" Jul 14 21:54:36.956485 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.956447 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c"} err="failed to get container status \"fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c\": rpc error: code = NotFound desc = could not find container \"fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c\": container with ID starting with fe40064f9e4b9129c01863a5b24173f9fc1420b375286f78f94e43dfbba4e51c not found: ID does not exist" Jul 14 21:54:36.968331 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.968298 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88df566fb-ld76v"] Jul 14 21:54:36.971609 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:36.971587 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-88df566fb-ld76v"] Jul 14 21:54:38.808498 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:38.808464 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3deefea7-ba2d-47be-bb21-91c8e31e4da8" path="/var/lib/kubelet/pods/3deefea7-ba2d-47be-bb21-91c8e31e4da8/volumes" Jul 14 21:54:50.397969 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.397928 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4"] Jul 14 21:54:50.398459 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.398359 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3deefea7-ba2d-47be-bb21-91c8e31e4da8" containerName="console" Jul 14 21:54:50.398459 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.398378 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3deefea7-ba2d-47be-bb21-91c8e31e4da8" containerName="console" Jul 14 21:54:50.398573 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.398477 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3deefea7-ba2d-47be-bb21-91c8e31e4da8" containerName="console" Jul 14 21:54:50.401333 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.401312 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.404600 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.404580 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 21:54:50.404883 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.404866 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zr7ln\"" Jul 14 21:54:50.405636 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.405624 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 21:54:50.423204 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.423180 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4"] Jul 14 21:54:50.523948 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.523912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.523948 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.523954 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.524186 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.523980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st49g\" (UniqueName: \"kubernetes.io/projected/7bc4509f-a002-4ab2-885b-96f091d8958e-kube-api-access-st49g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.625074 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.625041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st49g\" (UniqueName: \"kubernetes.io/projected/7bc4509f-a002-4ab2-885b-96f091d8958e-kube-api-access-st49g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.625243 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.625142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.625243 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.625179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.625565 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.625546 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.625601 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.625556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.634260 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.634239 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st49g\" (UniqueName: \"kubernetes.io/projected/7bc4509f-a002-4ab2-885b-96f091d8958e-kube-api-access-st49g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.710067 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.709969 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:54:50.832726 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:50.832655 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4"] Jul 14 21:54:50.835200 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:54:50.835166 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc4509f_a002_4ab2_885b_96f091d8958e.slice/crio-2e29975addb8662d798a19c648dda0a4245518184633399e86b9e3c42c21502b WatchSource:0}: Error finding container 2e29975addb8662d798a19c648dda0a4245518184633399e86b9e3c42c21502b: Status 404 returned error can't find the container with id 2e29975addb8662d798a19c648dda0a4245518184633399e86b9e3c42c21502b Jul 14 21:54:51.002407 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:51.002322 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" event={"ID":"7bc4509f-a002-4ab2-885b-96f091d8958e","Type":"ContainerStarted","Data":"2e29975addb8662d798a19c648dda0a4245518184633399e86b9e3c42c21502b"} Jul 14 21:54:57.029507 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:57.029466 2562 generic.go:356] "Generic (PLEG): container finished" podID="7bc4509f-a002-4ab2-885b-96f091d8958e" containerID="62839e5230098e3dda640b4fdbdfb2187f5cf3e1a996ae9bab3b5bac8cd8a136" exitCode=0 Jul 14 21:54:57.029879 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:54:57.029538 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" event={"ID":"7bc4509f-a002-4ab2-885b-96f091d8958e","Type":"ContainerDied","Data":"62839e5230098e3dda640b4fdbdfb2187f5cf3e1a996ae9bab3b5bac8cd8a136"} Jul 14 21:55:00.042793 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:00.042758 2562 generic.go:356] "Generic (PLEG): container finished" podID="7bc4509f-a002-4ab2-885b-96f091d8958e" containerID="1712dd1fc1d2111c8617f23d0f3b8f2b7ac7c05dc76c1a123c7c432b4de4700e" exitCode=0 Jul 14 21:55:00.043290 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:00.042822 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" event={"ID":"7bc4509f-a002-4ab2-885b-96f091d8958e","Type":"ContainerDied","Data":"1712dd1fc1d2111c8617f23d0f3b8f2b7ac7c05dc76c1a123c7c432b4de4700e"} Jul 14 21:55:06.071160 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:06.071125 2562 generic.go:356] "Generic (PLEG): container finished" podID="7bc4509f-a002-4ab2-885b-96f091d8958e" containerID="615db4778e6f05c3d26be7d39d33306d7324bf02f713161203ec877e3b22eb4a" exitCode=0 Jul 14 21:55:06.071565 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:06.071173 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" event={"ID":"7bc4509f-a002-4ab2-885b-96f091d8958e","Type":"ContainerDied","Data":"615db4778e6f05c3d26be7d39d33306d7324bf02f713161203ec877e3b22eb4a"} Jul 14 21:55:07.207428 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:07.207399 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:55:07.255004 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:07.254977 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-util\") pod \"7bc4509f-a002-4ab2-885b-96f091d8958e\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " Jul 14 21:55:07.255174 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:07.255068 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-bundle\") pod \"7bc4509f-a002-4ab2-885b-96f091d8958e\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " Jul 14 21:55:07.255174 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:07.255099 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st49g\" (UniqueName: \"kubernetes.io/projected/7bc4509f-a002-4ab2-885b-96f091d8958e-kube-api-access-st49g\") pod \"7bc4509f-a002-4ab2-885b-96f091d8958e\" (UID: \"7bc4509f-a002-4ab2-885b-96f091d8958e\") " Jul 14 21:55:07.255660 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:07.255634 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-bundle" (OuterVolumeSpecName: "bundle") pod "7bc4509f-a002-4ab2-885b-96f091d8958e" (UID: "7bc4509f-a002-4ab2-885b-96f091d8958e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:55:07.257222 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:07.257194 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc4509f-a002-4ab2-885b-96f091d8958e-kube-api-access-st49g" (OuterVolumeSpecName: "kube-api-access-st49g") pod "7bc4509f-a002-4ab2-885b-96f091d8958e" (UID: "7bc4509f-a002-4ab2-885b-96f091d8958e"). InnerVolumeSpecName "kube-api-access-st49g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:55:07.260053 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:07.260008 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-util" (OuterVolumeSpecName: "util") pod "7bc4509f-a002-4ab2-885b-96f091d8958e" (UID: "7bc4509f-a002-4ab2-885b-96f091d8958e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:55:07.355605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:07.355522 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:55:07.355605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:07.355547 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st49g\" (UniqueName: \"kubernetes.io/projected/7bc4509f-a002-4ab2-885b-96f091d8958e-kube-api-access-st49g\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:55:07.355605 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:07.355559 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7bc4509f-a002-4ab2-885b-96f091d8958e-util\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:55:08.080354 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:08.080323 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" Jul 14 21:55:08.080354 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:08.080330 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hmnd4" event={"ID":"7bc4509f-a002-4ab2-885b-96f091d8958e","Type":"ContainerDied","Data":"2e29975addb8662d798a19c648dda0a4245518184633399e86b9e3c42c21502b"} Jul 14 21:55:08.080569 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:08.080378 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e29975addb8662d798a19c648dda0a4245518184633399e86b9e3c42c21502b" Jul 14 21:55:13.454405 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.454369 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d"] Jul 14 21:55:13.454832 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.454779 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc4509f-a002-4ab2-885b-96f091d8958e" containerName="util" Jul 14 21:55:13.454832 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.454798 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc4509f-a002-4ab2-885b-96f091d8958e" containerName="util" Jul 14 21:55:13.454832 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.454809 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc4509f-a002-4ab2-885b-96f091d8958e" containerName="extract" Jul 14 21:55:13.454832 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.454814 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc4509f-a002-4ab2-885b-96f091d8958e" containerName="extract" Jul 14 21:55:13.454832 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.454830 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc4509f-a002-4ab2-885b-96f091d8958e" containerName="pull" Jul 14 21:55:13.454832 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.454837 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc4509f-a002-4ab2-885b-96f091d8958e" containerName="pull" Jul 14 21:55:13.455040 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.454887 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc4509f-a002-4ab2-885b-96f091d8958e" containerName="extract" Jul 14 21:55:13.496387 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.496359 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" Jul 14 21:55:13.499142 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.499117 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-x6bjw\"" Jul 14 21:55:13.499309 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.499155 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d"] Jul 14 21:55:13.499713 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.499694 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 14 21:55:13.504628 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.504601 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:55:13.600081 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.600039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8lf4d\" (UID: \"3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" Jul 14 21:55:13.600263 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.600149 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnl94\" (UniqueName: \"kubernetes.io/projected/3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7-kube-api-access-tnl94\") pod \"cert-manager-operator-controller-manager-55c444bf86-8lf4d\" (UID: \"3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" Jul 14 21:55:13.700682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.700644 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8lf4d\" (UID: \"3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" Jul 14 21:55:13.700837 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.700722 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnl94\" (UniqueName: \"kubernetes.io/projected/3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7-kube-api-access-tnl94\") pod \"cert-manager-operator-controller-manager-55c444bf86-8lf4d\" (UID: \"3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" Jul 14 21:55:13.701106 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.701071 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8lf4d\" (UID: \"3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" Jul 14 21:55:13.708974 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.708915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnl94\" (UniqueName: \"kubernetes.io/projected/3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7-kube-api-access-tnl94\") pod \"cert-manager-operator-controller-manager-55c444bf86-8lf4d\" (UID: \"3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" Jul 14 21:55:13.806373 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.806341 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" Jul 14 21:55:13.961283 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:13.961199 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d"] Jul 14 21:55:13.965216 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:55:13.965187 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e317ba2_19b0_4d22_83d6_4c3c11a9f1b7.slice/crio-1677611ac9b419439a2319c15c3e39a57f50a16b03626c3e10322de1ae51acc4 WatchSource:0}: Error finding container 1677611ac9b419439a2319c15c3e39a57f50a16b03626c3e10322de1ae51acc4: Status 404 returned error can't find the container with id 1677611ac9b419439a2319c15c3e39a57f50a16b03626c3e10322de1ae51acc4 Jul 14 21:55:14.105192 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:14.105158 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" event={"ID":"3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7","Type":"ContainerStarted","Data":"1677611ac9b419439a2319c15c3e39a57f50a16b03626c3e10322de1ae51acc4"} Jul 14 21:55:17.119933 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:17.119902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" event={"ID":"3e317ba2-19b0-4d22-83d6-4c3c11a9f1b7","Type":"ContainerStarted","Data":"37a2582c8fb53c61fb35a69a7e85857f5ca53a72451df4d35eb996d0e067941c"} Jul 14 21:55:17.155899 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:17.155841 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8lf4d" podStartSLOduration=1.761007024 podStartE2EDuration="4.155826042s" totalImagesPullingTime="2.394819018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:55:17.154003444 +0000 UTC m=+252.003473503" watchObservedRunningTime="2026-07-14 21:55:17.155826042 +0000 UTC m=+252.005296080" Jul 14 21:55:18.205375 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.205338 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7"] Jul 14 21:55:18.226084 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.226052 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.227037 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.226988 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7"] Jul 14 21:55:18.229187 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.229165 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zr7ln\"" Jul 14 21:55:18.229518 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.229503 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 21:55:18.230374 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.230352 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 21:55:18.335400 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.335359 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqpnx\" (UniqueName: \"kubernetes.io/projected/ecf52014-4c9d-4ff3-a036-502a251dfe5c-kube-api-access-gqpnx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.335591 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.335420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.335591 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.335473 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.436767 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.436731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.436973 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.436794 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqpnx\" (UniqueName: \"kubernetes.io/projected/ecf52014-4c9d-4ff3-a036-502a251dfe5c-kube-api-access-gqpnx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.436973 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.436858 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.437174 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.437149 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.437271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.437240 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.446422 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.446396 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqpnx\" (UniqueName: \"kubernetes.io/projected/ecf52014-4c9d-4ff3-a036-502a251dfe5c-kube-api-access-gqpnx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.535930 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.535894 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:18.689126 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:18.689097 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7"] Jul 14 21:55:18.690907 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:55:18.690876 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf52014_4c9d_4ff3_a036_502a251dfe5c.slice/crio-4f56476aec7307f6342e3e40aac3af3682417a3d67685edb8a31a02c1ffe2f3f WatchSource:0}: Error finding container 4f56476aec7307f6342e3e40aac3af3682417a3d67685edb8a31a02c1ffe2f3f: Status 404 returned error can't find the container with id 4f56476aec7307f6342e3e40aac3af3682417a3d67685edb8a31a02c1ffe2f3f Jul 14 21:55:19.129840 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:19.129760 2562 generic.go:356] "Generic (PLEG): container finished" podID="ecf52014-4c9d-4ff3-a036-502a251dfe5c" containerID="7b13ad37a7f91c7e4493baec4fe38129f248923286dd3d5d8ba939aa5804d9b4" exitCode=0 Jul 14 21:55:19.129840 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:19.129805 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" event={"ID":"ecf52014-4c9d-4ff3-a036-502a251dfe5c","Type":"ContainerDied","Data":"7b13ad37a7f91c7e4493baec4fe38129f248923286dd3d5d8ba939aa5804d9b4"} Jul 14 21:55:19.130041 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:19.129841 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" event={"ID":"ecf52014-4c9d-4ff3-a036-502a251dfe5c","Type":"ContainerStarted","Data":"4f56476aec7307f6342e3e40aac3af3682417a3d67685edb8a31a02c1ffe2f3f"} Jul 14 21:55:20.365537 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.365497 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vjktk"] Jul 14 21:55:20.369867 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.369837 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" Jul 14 21:55:20.372518 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.372496 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 21:55:20.373602 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.373462 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vcqwx\"" Jul 14 21:55:20.373602 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.373563 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 21:55:20.378736 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.378693 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vjktk"] Jul 14 21:55:20.453457 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.453420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d0d6be8-974a-4dcf-93d1-c61633bcbede-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vjktk\" (UID: \"4d0d6be8-974a-4dcf-93d1-c61633bcbede\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" Jul 14 21:55:20.453627 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.453478 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4z6w\" (UniqueName: \"kubernetes.io/projected/4d0d6be8-974a-4dcf-93d1-c61633bcbede-kube-api-access-v4z6w\") pod \"cert-manager-cainjector-559cdd8c87-vjktk\" (UID: \"4d0d6be8-974a-4dcf-93d1-c61633bcbede\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" Jul 14 21:55:20.554713 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.554676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d0d6be8-974a-4dcf-93d1-c61633bcbede-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vjktk\" (UID: \"4d0d6be8-974a-4dcf-93d1-c61633bcbede\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" Jul 14 21:55:20.554886 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.554731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4z6w\" (UniqueName: \"kubernetes.io/projected/4d0d6be8-974a-4dcf-93d1-c61633bcbede-kube-api-access-v4z6w\") pod \"cert-manager-cainjector-559cdd8c87-vjktk\" (UID: \"4d0d6be8-974a-4dcf-93d1-c61633bcbede\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" Jul 14 21:55:20.563635 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.563605 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d0d6be8-974a-4dcf-93d1-c61633bcbede-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vjktk\" (UID: \"4d0d6be8-974a-4dcf-93d1-c61633bcbede\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" Jul 14 21:55:20.563784 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.563737 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4z6w\" (UniqueName: \"kubernetes.io/projected/4d0d6be8-974a-4dcf-93d1-c61633bcbede-kube-api-access-v4z6w\") pod \"cert-manager-cainjector-559cdd8c87-vjktk\" (UID: \"4d0d6be8-974a-4dcf-93d1-c61633bcbede\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" Jul 14 21:55:20.681748 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:20.681665 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" Jul 14 21:55:21.084914 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:21.084879 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vjktk"] Jul 14 21:55:21.087522 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:55:21.087487 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d0d6be8_974a_4dcf_93d1_c61633bcbede.slice/crio-f0522ee076e0969e5439ea79c12170c0af3fb1d7e52b2c09273115ba5f451f1d WatchSource:0}: Error finding container f0522ee076e0969e5439ea79c12170c0af3fb1d7e52b2c09273115ba5f451f1d: Status 404 returned error can't find the container with id f0522ee076e0969e5439ea79c12170c0af3fb1d7e52b2c09273115ba5f451f1d Jul 14 21:55:21.140427 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:21.140398 2562 generic.go:356] "Generic (PLEG): container finished" podID="ecf52014-4c9d-4ff3-a036-502a251dfe5c" containerID="999449e409a7dac9058d92c81b68dec6d0bf0c9e26bcf28a22a06ea7a7bc8bef" exitCode=0 Jul 14 21:55:21.140577 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:21.140489 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" event={"ID":"ecf52014-4c9d-4ff3-a036-502a251dfe5c","Type":"ContainerDied","Data":"999449e409a7dac9058d92c81b68dec6d0bf0c9e26bcf28a22a06ea7a7bc8bef"} Jul 14 21:55:21.141789 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:21.141750 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" event={"ID":"4d0d6be8-974a-4dcf-93d1-c61633bcbede","Type":"ContainerStarted","Data":"f0522ee076e0969e5439ea79c12170c0af3fb1d7e52b2c09273115ba5f451f1d"} Jul 14 21:55:22.154435 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:22.154398 2562 generic.go:356] "Generic (PLEG): container finished" podID="ecf52014-4c9d-4ff3-a036-502a251dfe5c" containerID="44db837b11bbca917e0c10def6142f32eaa191c202df5edc4a1a94650cd90bd2" exitCode=0 Jul 14 21:55:22.154807 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:22.154469 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" event={"ID":"ecf52014-4c9d-4ff3-a036-502a251dfe5c","Type":"ContainerDied","Data":"44db837b11bbca917e0c10def6142f32eaa191c202df5edc4a1a94650cd90bd2"} Jul 14 21:55:23.297320 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:23.297296 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:23.380103 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:23.380075 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-bundle\") pod \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " Jul 14 21:55:23.380289 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:23.380181 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqpnx\" (UniqueName: \"kubernetes.io/projected/ecf52014-4c9d-4ff3-a036-502a251dfe5c-kube-api-access-gqpnx\") pod \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " Jul 14 21:55:23.380289 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:23.380225 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-util\") pod \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\" (UID: \"ecf52014-4c9d-4ff3-a036-502a251dfe5c\") " Jul 14 21:55:23.380669 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:23.380631 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-bundle" (OuterVolumeSpecName: "bundle") pod "ecf52014-4c9d-4ff3-a036-502a251dfe5c" (UID: "ecf52014-4c9d-4ff3-a036-502a251dfe5c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:55:23.382587 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:23.382557 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf52014-4c9d-4ff3-a036-502a251dfe5c-kube-api-access-gqpnx" (OuterVolumeSpecName: "kube-api-access-gqpnx") pod "ecf52014-4c9d-4ff3-a036-502a251dfe5c" (UID: "ecf52014-4c9d-4ff3-a036-502a251dfe5c"). InnerVolumeSpecName "kube-api-access-gqpnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:55:23.387646 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:23.387609 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-util" (OuterVolumeSpecName: "util") pod "ecf52014-4c9d-4ff3-a036-502a251dfe5c" (UID: "ecf52014-4c9d-4ff3-a036-502a251dfe5c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:55:23.481320 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:23.481236 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:55:23.481320 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:23.481276 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqpnx\" (UniqueName: \"kubernetes.io/projected/ecf52014-4c9d-4ff3-a036-502a251dfe5c-kube-api-access-gqpnx\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:55:23.481320 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:23.481291 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecf52014-4c9d-4ff3-a036-502a251dfe5c-util\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:55:24.166297 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.166257 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" event={"ID":"ecf52014-4c9d-4ff3-a036-502a251dfe5c","Type":"ContainerDied","Data":"4f56476aec7307f6342e3e40aac3af3682417a3d67685edb8a31a02c1ffe2f3f"} Jul 14 21:55:24.166297 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.166296 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6khf7" Jul 14 21:55:24.166515 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.166311 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f56476aec7307f6342e3e40aac3af3682417a3d67685edb8a31a02c1ffe2f3f" Jul 14 21:55:24.874714 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.874677 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m6t56"] Jul 14 21:55:24.875203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.875062 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf52014-4c9d-4ff3-a036-502a251dfe5c" containerName="pull" Jul 14 21:55:24.875203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.875078 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf52014-4c9d-4ff3-a036-502a251dfe5c" containerName="pull" Jul 14 21:55:24.875203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.875093 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf52014-4c9d-4ff3-a036-502a251dfe5c" containerName="util" Jul 14 21:55:24.875203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.875099 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf52014-4c9d-4ff3-a036-502a251dfe5c" containerName="util" Jul 14 21:55:24.875203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.875110 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf52014-4c9d-4ff3-a036-502a251dfe5c" containerName="extract" Jul 14 21:55:24.875203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.875116 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf52014-4c9d-4ff3-a036-502a251dfe5c" containerName="extract" Jul 14 21:55:24.875203 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.875187 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf52014-4c9d-4ff3-a036-502a251dfe5c" containerName="extract" Jul 14 21:55:24.877260 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.877236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" Jul 14 21:55:24.881901 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.881855 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-28nw2\"" Jul 14 21:55:24.897156 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.897111 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m6t56"] Jul 14 21:55:24.992986 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.992945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmf2w\" (UniqueName: \"kubernetes.io/projected/5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6-kube-api-access-tmf2w\") pod \"cert-manager-webhook-6486ffd997-m6t56\" (UID: \"5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" Jul 14 21:55:24.993195 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:24.993071 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m6t56\" (UID: \"5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" Jul 14 21:55:25.093981 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:25.093950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmf2w\" (UniqueName: \"kubernetes.io/projected/5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6-kube-api-access-tmf2w\") pod \"cert-manager-webhook-6486ffd997-m6t56\" (UID: \"5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" Jul 14 21:55:25.094168 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:25.093991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m6t56\" (UID: \"5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" Jul 14 21:55:25.101932 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:25.101907 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m6t56\" (UID: \"5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" Jul 14 21:55:25.102104 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:25.102075 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmf2w\" (UniqueName: \"kubernetes.io/projected/5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6-kube-api-access-tmf2w\") pod \"cert-manager-webhook-6486ffd997-m6t56\" (UID: \"5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" Jul 14 21:55:25.187969 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:25.187942 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" Jul 14 21:55:25.343155 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:25.343110 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m6t56"] Jul 14 21:55:25.349978 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:55:25.349927 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f0ac9fd_1f6c_4c7b_b9e2_619741b3c5e6.slice/crio-4848450e89b04f1fd4d2904a1bca81f69d9e65a488be9bee772181924e25791d WatchSource:0}: Error finding container 4848450e89b04f1fd4d2904a1bca81f69d9e65a488be9bee772181924e25791d: Status 404 returned error can't find the container with id 4848450e89b04f1fd4d2904a1bca81f69d9e65a488be9bee772181924e25791d Jul 14 21:55:26.175412 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:26.175372 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" event={"ID":"4d0d6be8-974a-4dcf-93d1-c61633bcbede","Type":"ContainerStarted","Data":"62ae1f194e08f0474fe47b0ba88b0f31e9a920d8381f020a2785c64f527de8b9"} Jul 14 21:55:26.176866 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:26.176843 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" event={"ID":"5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6","Type":"ContainerStarted","Data":"783e522ff4a7f7d272ba0454fabe99d3f16653f375e1be4d09380801de0d7473"} Jul 14 21:55:26.176997 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:26.176874 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" event={"ID":"5f0ac9fd-1f6c-4c7b-b9e2-619741b3c5e6","Type":"ContainerStarted","Data":"4848450e89b04f1fd4d2904a1bca81f69d9e65a488be9bee772181924e25791d"} Jul 14 21:55:26.176997 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:26.176950 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" Jul 14 21:55:26.194606 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:26.194553 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vjktk" podStartSLOduration=2.145916505 podStartE2EDuration="6.194538199s" totalImagesPullingTime="4.048621694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:55:26.192247454 +0000 UTC m=+261.041717492" watchObservedRunningTime="2026-07-14 21:55:26.194538199 +0000 UTC m=+261.044008238" Jul 14 21:55:26.218383 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:26.218330 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" podStartSLOduration=2.218312315 podStartE2EDuration="2.218312315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:55:26.216534686 +0000 UTC m=+261.066004738" watchObservedRunningTime="2026-07-14 21:55:26.218312315 +0000 UTC m=+261.067782353" Jul 14 21:55:27.505582 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.505473 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s8mvf"] Jul 14 21:55:27.507578 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.507561 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s8mvf" Jul 14 21:55:27.511936 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.511915 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-67csh\"" Jul 14 21:55:27.526060 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.526038 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s8mvf"] Jul 14 21:55:27.612991 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.612952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35eff7c2-6123-47fe-aad3-7c690a216c84-bound-sa-token\") pod \"cert-manager-55b4d954d5-s8mvf\" (UID: \"35eff7c2-6123-47fe-aad3-7c690a216c84\") " pod="cert-manager/cert-manager-55b4d954d5-s8mvf" Jul 14 21:55:27.613200 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.613000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24c9h\" (UniqueName: \"kubernetes.io/projected/35eff7c2-6123-47fe-aad3-7c690a216c84-kube-api-access-24c9h\") pod \"cert-manager-55b4d954d5-s8mvf\" (UID: \"35eff7c2-6123-47fe-aad3-7c690a216c84\") " pod="cert-manager/cert-manager-55b4d954d5-s8mvf" Jul 14 21:55:27.714112 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.714069 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35eff7c2-6123-47fe-aad3-7c690a216c84-bound-sa-token\") pod \"cert-manager-55b4d954d5-s8mvf\" (UID: \"35eff7c2-6123-47fe-aad3-7c690a216c84\") " pod="cert-manager/cert-manager-55b4d954d5-s8mvf" Jul 14 21:55:27.714300 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.714127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24c9h\" (UniqueName: \"kubernetes.io/projected/35eff7c2-6123-47fe-aad3-7c690a216c84-kube-api-access-24c9h\") pod \"cert-manager-55b4d954d5-s8mvf\" (UID: \"35eff7c2-6123-47fe-aad3-7c690a216c84\") " pod="cert-manager/cert-manager-55b4d954d5-s8mvf" Jul 14 21:55:27.722501 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.722466 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35eff7c2-6123-47fe-aad3-7c690a216c84-bound-sa-token\") pod \"cert-manager-55b4d954d5-s8mvf\" (UID: \"35eff7c2-6123-47fe-aad3-7c690a216c84\") " pod="cert-manager/cert-manager-55b4d954d5-s8mvf" Jul 14 21:55:27.722646 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.722629 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24c9h\" (UniqueName: \"kubernetes.io/projected/35eff7c2-6123-47fe-aad3-7c690a216c84-kube-api-access-24c9h\") pod \"cert-manager-55b4d954d5-s8mvf\" (UID: \"35eff7c2-6123-47fe-aad3-7c690a216c84\") " pod="cert-manager/cert-manager-55b4d954d5-s8mvf" Jul 14 21:55:27.816625 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.816596 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s8mvf" Jul 14 21:55:27.949332 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:27.949279 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s8mvf"] Jul 14 21:55:27.951647 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:55:27.951612 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35eff7c2_6123_47fe_aad3_7c690a216c84.slice/crio-9624eb8ed82e07b6e186fa6116d43d8d23ff54576dfe1121cb6e26e7dcd493ed WatchSource:0}: Error finding container 9624eb8ed82e07b6e186fa6116d43d8d23ff54576dfe1121cb6e26e7dcd493ed: Status 404 returned error can't find the container with id 9624eb8ed82e07b6e186fa6116d43d8d23ff54576dfe1121cb6e26e7dcd493ed Jul 14 21:55:28.186652 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:28.186552 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s8mvf" event={"ID":"35eff7c2-6123-47fe-aad3-7c690a216c84","Type":"ContainerStarted","Data":"b106923448bd9c0d6aa1d10f55c3880ef0f743b8ecfe416fa0190c5f94578101"} Jul 14 21:55:28.186652 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:28.186596 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s8mvf" event={"ID":"35eff7c2-6123-47fe-aad3-7c690a216c84","Type":"ContainerStarted","Data":"9624eb8ed82e07b6e186fa6116d43d8d23ff54576dfe1121cb6e26e7dcd493ed"} Jul 14 21:55:28.203111 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:28.202959 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-s8mvf" podStartSLOduration=1.202941992 podStartE2EDuration="1.202941992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:55:28.202777806 +0000 UTC m=+263.052247865" watchObservedRunningTime="2026-07-14 21:55:28.202941992 +0000 UTC m=+263.052412043" Jul 14 21:55:32.183521 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:32.183487 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m6t56" Jul 14 21:55:38.275415 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.275036 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r"] Jul 14 21:55:38.280227 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.280196 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.283449 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.283426 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zr7ln\"" Jul 14 21:55:38.283577 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.283537 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 21:55:38.284537 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.284516 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 21:55:38.292972 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.292945 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r"] Jul 14 21:55:38.407712 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.407644 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.407873 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.407776 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzd2v\" (UniqueName: \"kubernetes.io/projected/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-kube-api-access-kzd2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.407873 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.407858 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.508444 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.508398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.508444 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.508445 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.508720 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.508497 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzd2v\" (UniqueName: \"kubernetes.io/projected/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-kube-api-access-kzd2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.508893 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.508868 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.508960 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.508882 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.516905 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.516879 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzd2v\" (UniqueName: \"kubernetes.io/projected/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-kube-api-access-kzd2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.590609 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.590528 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:38.723842 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:38.723803 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r"] Jul 14 21:55:38.727328 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:55:38.727300 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2538a1_e4bc_4dbc_bbc2_3543e79eaec2.slice/crio-388e3cae88ab26a08c27e50265ee88003ccfb24f9f615a94510797261b8eb906 WatchSource:0}: Error finding container 388e3cae88ab26a08c27e50265ee88003ccfb24f9f615a94510797261b8eb906: Status 404 returned error can't find the container with id 388e3cae88ab26a08c27e50265ee88003ccfb24f9f615a94510797261b8eb906 Jul 14 21:55:39.235347 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:39.235310 2562 generic.go:356] "Generic (PLEG): container finished" podID="4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" containerID="21f59e823dd9612113fc1886f8de9e1214a04da82067cdb3c50c1247eb7c01ad" exitCode=0 Jul 14 21:55:39.235524 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:39.235391 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" event={"ID":"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2","Type":"ContainerDied","Data":"21f59e823dd9612113fc1886f8de9e1214a04da82067cdb3c50c1247eb7c01ad"} Jul 14 21:55:39.235524 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:39.235439 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" event={"ID":"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2","Type":"ContainerStarted","Data":"388e3cae88ab26a08c27e50265ee88003ccfb24f9f615a94510797261b8eb906"} Jul 14 21:55:41.246713 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:41.246677 2562 generic.go:356] "Generic (PLEG): container finished" podID="4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" containerID="0236dc0f2655b3f9d6548cba61f9e550b1a6e933c5b740f9d8ead845db7def45" exitCode=0 Jul 14 21:55:41.247159 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:41.246753 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" event={"ID":"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2","Type":"ContainerDied","Data":"0236dc0f2655b3f9d6548cba61f9e550b1a6e933c5b740f9d8ead845db7def45"} Jul 14 21:55:42.253185 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:42.253148 2562 generic.go:356] "Generic (PLEG): container finished" podID="4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" containerID="404f4bec6d9ee04910db401af74c88b257559b812649df348eff01bb87c305e9" exitCode=0 Jul 14 21:55:42.253578 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:42.253238 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" event={"ID":"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2","Type":"ContainerDied","Data":"404f4bec6d9ee04910db401af74c88b257559b812649df348eff01bb87c305e9"} Jul 14 21:55:43.388116 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:43.388091 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:43.551800 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:43.551764 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzd2v\" (UniqueName: \"kubernetes.io/projected/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-kube-api-access-kzd2v\") pod \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " Jul 14 21:55:43.551964 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:43.551821 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-bundle\") pod \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " Jul 14 21:55:43.551964 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:43.551881 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-util\") pod \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\" (UID: \"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2\") " Jul 14 21:55:43.552563 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:43.552527 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-bundle" (OuterVolumeSpecName: "bundle") pod "4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" (UID: "4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:55:43.554180 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:43.554159 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-kube-api-access-kzd2v" (OuterVolumeSpecName: "kube-api-access-kzd2v") pod "4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" (UID: "4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2"). InnerVolumeSpecName "kube-api-access-kzd2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:55:43.560039 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:43.559997 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-util" (OuterVolumeSpecName: "util") pod "4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" (UID: "4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:55:43.652705 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:43.652654 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-util\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:55:43.652705 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:43.652698 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzd2v\" (UniqueName: \"kubernetes.io/projected/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-kube-api-access-kzd2v\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:55:43.652705 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:43.652708 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:55:44.264091 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:44.264060 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" Jul 14 21:55:44.264278 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:44.264060 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm9k4r" event={"ID":"4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2","Type":"ContainerDied","Data":"388e3cae88ab26a08c27e50265ee88003ccfb24f9f615a94510797261b8eb906"} Jul 14 21:55:44.264278 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:44.264201 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="388e3cae88ab26a08c27e50265ee88003ccfb24f9f615a94510797261b8eb906" Jul 14 21:55:56.082186 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.082148 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27"] Jul 14 21:55:56.082671 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.082607 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" containerName="util" Jul 14 21:55:56.082671 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.082626 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" containerName="util" Jul 14 21:55:56.082671 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.082644 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" containerName="pull" Jul 14 21:55:56.082671 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.082653 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" containerName="pull" Jul 14 21:55:56.082671 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.082666 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" containerName="extract" Jul 14 21:55:56.082839 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.082679 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" containerName="extract" Jul 14 21:55:56.082839 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.082762 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2538a1-e4bc-4dbc-bbc2-3543e79eaec2" containerName="extract" Jul 14 21:55:56.129351 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.129317 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27"] Jul 14 21:55:56.129501 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.129421 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.132534 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.132513 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 21:55:56.133419 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.133400 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 21:55:56.133580 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.133563 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 21:55:56.133580 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.133576 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 21:55:56.133731 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.133714 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8svd6\"" Jul 14 21:55:56.261772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.261734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/257db741-246e-4426-818a-54585d5ddb7b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.261772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.261775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rz64\" (UniqueName: \"kubernetes.io/projected/257db741-246e-4426-818a-54585d5ddb7b-kube-api-access-9rz64\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.262005 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.261810 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/257db741-246e-4426-818a-54585d5ddb7b-manifests\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.262005 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.261911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/257db741-246e-4426-818a-54585d5ddb7b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.262005 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.261953 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/257db741-246e-4426-818a-54585d5ddb7b-tmp\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.362711 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.362617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/257db741-246e-4426-818a-54585d5ddb7b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.362711 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.362670 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/257db741-246e-4426-818a-54585d5ddb7b-tmp\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.362711 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.362705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/257db741-246e-4426-818a-54585d5ddb7b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.362982 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.362733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rz64\" (UniqueName: \"kubernetes.io/projected/257db741-246e-4426-818a-54585d5ddb7b-kube-api-access-9rz64\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.362982 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.362871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/257db741-246e-4426-818a-54585d5ddb7b-manifests\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.363271 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.363238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/257db741-246e-4426-818a-54585d5ddb7b-manifests\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.365053 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.364996 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/257db741-246e-4426-818a-54585d5ddb7b-tmp\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.365180 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.365125 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/257db741-246e-4426-818a-54585d5ddb7b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.365329 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.365311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/257db741-246e-4426-818a-54585d5ddb7b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.372389 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.372350 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rz64\" (UniqueName: \"kubernetes.io/projected/257db741-246e-4426-818a-54585d5ddb7b-kube-api-access-9rz64\") pod \"opendatahub-operator-controller-manager-5c99649887-rgv27\" (UID: \"257db741-246e-4426-818a-54585d5ddb7b\") " pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.439373 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.439341 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:55:56.648092 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:56.648066 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27"] Jul 14 21:55:56.650638 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:55:56.650607 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257db741_246e_4426_818a_54585d5ddb7b.slice/crio-0f87499eb8ba08492dae65643fbb50ccacc180435a7f8c0bdf91d75be3f6934b WatchSource:0}: Error finding container 0f87499eb8ba08492dae65643fbb50ccacc180435a7f8c0bdf91d75be3f6934b: Status 404 returned error can't find the container with id 0f87499eb8ba08492dae65643fbb50ccacc180435a7f8c0bdf91d75be3f6934b Jul 14 21:55:57.322259 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:55:57.322223 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" event={"ID":"257db741-246e-4426-818a-54585d5ddb7b","Type":"ContainerStarted","Data":"0f87499eb8ba08492dae65643fbb50ccacc180435a7f8c0bdf91d75be3f6934b"} Jul 14 21:56:00.339923 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:00.339881 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" event={"ID":"257db741-246e-4426-818a-54585d5ddb7b","Type":"ContainerStarted","Data":"e276cad56e3860b7eee93921610a43dc52fa34e11d26544d4911fe8830ba491c"} Jul 14 21:56:01.346276 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:01.346237 2562 generic.go:356] "Generic (PLEG): container finished" podID="257db741-246e-4426-818a-54585d5ddb7b" containerID="e276cad56e3860b7eee93921610a43dc52fa34e11d26544d4911fe8830ba491c" exitCode=0 Jul 14 21:56:01.346656 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:01.346328 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" event={"ID":"257db741-246e-4426-818a-54585d5ddb7b","Type":"ContainerDied","Data":"e276cad56e3860b7eee93921610a43dc52fa34e11d26544d4911fe8830ba491c"} Jul 14 21:56:02.352992 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:02.352958 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" event={"ID":"257db741-246e-4426-818a-54585d5ddb7b","Type":"ContainerStarted","Data":"554d54730bf21519c6d14dfec81581595db7c8a48bfb3e8ed584d291b9c62ad1"} Jul 14 21:56:02.353372 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:02.353160 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:56:02.387116 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:02.387059 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" podStartSLOduration=1.556683445 podStartE2EDuration="6.387039287s" totalImagesPullingTime="3.816251273s" totalInitContainerRuntime="1.014104569s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:55:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:56:02.384134739 +0000 UTC m=+297.233604779" watchObservedRunningTime="2026-07-14 21:56:02.387039287 +0000 UTC m=+297.236509319" Jul 14 21:56:06.701635 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:06.701608 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 21:56:13.360044 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:13.359994 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c99649887-rgv27" Jul 14 21:56:24.127311 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.127272 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n"] Jul 14 21:56:24.132679 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.132656 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68"] Jul 14 21:56:24.132839 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.132727 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.135985 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.135962 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 21:56:24.136217 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.136200 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.136305 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.136264 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 21:56:24.137274 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.137261 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 21:56:24.137343 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.137271 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-h7vcx\"" Jul 14 21:56:24.139905 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.139891 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-f7fdj\"" Jul 14 21:56:24.140218 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.140197 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 21:56:24.140416 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.140399 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 21:56:24.140510 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.140469 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 21:56:24.147545 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.147523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n"] Jul 14 21:56:24.152952 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.152932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68"] Jul 14 21:56:24.165230 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165203 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedecead-83e4-425c-bac2-4d58a627cf2a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.165376 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f7825e5-d4b5-489f-a4c2-32ab490c384c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.165376 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165259 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.165376 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.165376 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n8f2\" (UniqueName: \"kubernetes.io/projected/6f7825e5-d4b5-489f-a4c2-32ab490c384c-kube-api-access-4n8f2\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.165376 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165347 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.165376 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165363 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.165654 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165412 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.165654 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165456 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.165654 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7825e5-d4b5-489f-a4c2-32ab490c384c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.165654 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.165654 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165588 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.165654 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.165858 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165672 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.165858 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165689 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.165858 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.165707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbr82\" (UniqueName: \"kubernetes.io/projected/bedecead-83e4-425c-bac2-4d58a627cf2a-kube-api-access-fbr82\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.266975 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.266942 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.267202 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.266987 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.267202 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.267202 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbr82\" (UniqueName: \"kubernetes.io/projected/bedecead-83e4-425c-bac2-4d58a627cf2a-kube-api-access-fbr82\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.267202 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedecead-83e4-425c-bac2-4d58a627cf2a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.267202 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f7825e5-d4b5-489f-a4c2-32ab490c384c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.267202 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.267532 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.267532 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n8f2\" (UniqueName: \"kubernetes.io/projected/6f7825e5-d4b5-489f-a4c2-32ab490c384c-kube-api-access-4n8f2\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.267532 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.267532 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267391 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.267532 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267423 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.267532 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.267532 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267469 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.267532 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.268015 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7825e5-d4b5-489f-a4c2-32ab490c384c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.268015 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.268015 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.268015 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267833 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.268015 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.267867 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedecead-83e4-425c-bac2-4d58a627cf2a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.268305 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.268110 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.268305 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.268204 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.270134 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.270109 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f7825e5-d4b5-489f-a4c2-32ab490c384c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.270247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.270208 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.270410 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.270389 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.270471 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.270412 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.270471 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.270394 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.270576 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.270506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7825e5-d4b5-489f-a4c2-32ab490c384c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.280643 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.280595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n8f2\" (UniqueName: \"kubernetes.io/projected/6f7825e5-d4b5-489f-a4c2-32ab490c384c-kube-api-access-4n8f2\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.280784 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.280676 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedecead-83e4-425c-bac2-4d58a627cf2a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.281120 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.281012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7825e5-d4b5-489f-a4c2-32ab490c384c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4wc68\" (UID: \"6f7825e5-d4b5-489f-a4c2-32ab490c384c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.281353 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.281336 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbr82\" (UniqueName: \"kubernetes.io/projected/bedecead-83e4-425c-bac2-4d58a627cf2a-kube-api-access-fbr82\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xc8n\" (UID: \"bedecead-83e4-425c-bac2-4d58a627cf2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.444580 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.444479 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:24.450792 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.450773 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:24.621934 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.621897 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68"] Jul 14 21:56:24.624788 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:56:24.624763 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7825e5_d4b5_489f_a4c2_32ab490c384c.slice/crio-ba1a0d09abb6498c0b51c65140152e93598f1014b2608e29799633744a282c86 WatchSource:0}: Error finding container ba1a0d09abb6498c0b51c65140152e93598f1014b2608e29799633744a282c86: Status 404 returned error can't find the container with id ba1a0d09abb6498c0b51c65140152e93598f1014b2608e29799633744a282c86 Jul 14 21:56:24.626599 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.626584 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 21:56:24.639018 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:24.638996 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n"] Jul 14 21:56:24.639872 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:56:24.639846 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedecead_83e4_425c_bac2_4d58a627cf2a.slice/crio-99d02080fbb1945480a52171dbb95d222e2450a4a651a6b19ce8d2abab7bc4d5 WatchSource:0}: Error finding container 99d02080fbb1945480a52171dbb95d222e2450a4a651a6b19ce8d2abab7bc4d5: Status 404 returned error can't find the container with id 99d02080fbb1945480a52171dbb95d222e2450a4a651a6b19ce8d2abab7bc4d5 Jul 14 21:56:25.464272 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.464232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" event={"ID":"bedecead-83e4-425c-bac2-4d58a627cf2a","Type":"ContainerStarted","Data":"99d02080fbb1945480a52171dbb95d222e2450a4a651a6b19ce8d2abab7bc4d5"} Jul 14 21:56:25.465480 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.465457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" event={"ID":"6f7825e5-d4b5-489f-a4c2-32ab490c384c","Type":"ContainerStarted","Data":"ba1a0d09abb6498c0b51c65140152e93598f1014b2608e29799633744a282c86"} Jul 14 21:56:25.742425 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.742343 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9"] Jul 14 21:56:25.747140 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.747116 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.749947 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.749919 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 21:56:25.750096 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.749966 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 21:56:25.751114 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.751094 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:56:25.751273 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.751256 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 21:56:25.751346 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.751255 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 21:56:25.751346 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.751330 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gtzrr\"" Jul 14 21:56:25.761049 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.759132 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9"] Jul 14 21:56:25.780951 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.780912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmmz8\" (UniqueName: \"kubernetes.io/projected/8728d6cf-3089-45f6-b92a-c716e967ade4-kube-api-access-gmmz8\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.781136 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.780958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8728d6cf-3089-45f6-b92a-c716e967ade4-cert\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.781136 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.781047 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8728d6cf-3089-45f6-b92a-c716e967ade4-manager-config\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.781213 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.781148 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8728d6cf-3089-45f6-b92a-c716e967ade4-metrics-cert\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.882393 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.882353 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmmz8\" (UniqueName: \"kubernetes.io/projected/8728d6cf-3089-45f6-b92a-c716e967ade4-kube-api-access-gmmz8\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.882589 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.882408 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8728d6cf-3089-45f6-b92a-c716e967ade4-cert\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.882589 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.882463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8728d6cf-3089-45f6-b92a-c716e967ade4-manager-config\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.882589 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.882523 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8728d6cf-3089-45f6-b92a-c716e967ade4-metrics-cert\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.883209 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.883181 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8728d6cf-3089-45f6-b92a-c716e967ade4-manager-config\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.885087 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.885066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8728d6cf-3089-45f6-b92a-c716e967ade4-metrics-cert\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.885167 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.885105 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8728d6cf-3089-45f6-b92a-c716e967ade4-cert\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:25.893138 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:25.893107 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmmz8\" (UniqueName: \"kubernetes.io/projected/8728d6cf-3089-45f6-b92a-c716e967ade4-kube-api-access-gmmz8\") pod \"lws-controller-manager-676d99795d-d8gv9\" (UID: \"8728d6cf-3089-45f6-b92a-c716e967ade4\") " pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:26.058651 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:26.058612 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:26.247511 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:26.247479 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9"] Jul 14 21:56:26.422391 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:56:26.422298 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8728d6cf_3089_45f6_b92a_c716e967ade4.slice/crio-d4555a0351c67e14b8880ef106b0f18339e803eb84c37fc8ec008584184df088 WatchSource:0}: Error finding container d4555a0351c67e14b8880ef106b0f18339e803eb84c37fc8ec008584184df088: Status 404 returned error can't find the container with id d4555a0351c67e14b8880ef106b0f18339e803eb84c37fc8ec008584184df088 Jul 14 21:56:26.474487 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:26.474436 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" event={"ID":"8728d6cf-3089-45f6-b92a-c716e967ade4","Type":"ContainerStarted","Data":"d4555a0351c67e14b8880ef106b0f18339e803eb84c37fc8ec008584184df088"} Jul 14 21:56:29.648850 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:29.648809 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 21:56:29.649167 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:29.648886 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 21:56:29.653212 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:29.653181 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 21:56:29.653295 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:29.653244 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 21:56:29.653295 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:29.653276 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 21:56:30.500311 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:30.500275 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" event={"ID":"8728d6cf-3089-45f6-b92a-c716e967ade4","Type":"ContainerStarted","Data":"35e153656aafeca9c89eec200517c19d5cc7a145124714e0e8acbd9df7ff19c2"} Jul 14 21:56:30.500511 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:30.500374 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:56:30.502048 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:30.502003 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" event={"ID":"6f7825e5-d4b5-489f-a4c2-32ab490c384c","Type":"ContainerStarted","Data":"f5632c1a9377b63daa2098774801e008868a1c5254c61114d1d5466f4c10b97d"} Jul 14 21:56:30.502180 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:30.502071 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:30.503833 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:30.503801 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" event={"ID":"bedecead-83e4-425c-bac2-4d58a627cf2a","Type":"ContainerStarted","Data":"f598eeafe9fd142a75aca51f7db3ae0be1f844e0c822824e538e2fab869ded46"} Jul 14 21:56:30.504165 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:30.504140 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" Jul 14 21:56:30.522857 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:30.522803 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" podStartSLOduration=2.24945648 podStartE2EDuration="5.522788678s" totalImagesPullingTime="3.273332198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:56:30.520719447 +0000 UTC m=+325.370189487" watchObservedRunningTime="2026-07-14 21:56:30.522788678 +0000 UTC m=+325.372258696" Jul 14 21:56:30.548008 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:30.547953 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wc68" podStartSLOduration=1.526098162 podStartE2EDuration="6.547938553s" totalImagesPullingTime="5.021840391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:56:30.545849408 +0000 UTC m=+325.395319453" watchObservedRunningTime="2026-07-14 21:56:30.547938553 +0000 UTC m=+325.397408591" Jul 14 21:56:30.576233 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:30.576180 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" podStartSLOduration=1.565120273 podStartE2EDuration="6.576161424s" totalImagesPullingTime="5.011041151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:56:30.574890594 +0000 UTC m=+325.424360634" watchObservedRunningTime="2026-07-14 21:56:30.576161424 +0000 UTC m=+325.425631462" Jul 14 21:56:31.445550 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:31.445511 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:31.449875 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:31.449848 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:31.510656 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:31.510615 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:31.511669 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:31.511647 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xc8n" Jul 14 21:56:41.512472 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:56:41.512435 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-676d99795d-d8gv9" Jul 14 21:57:33.489008 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.488881 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj"] Jul 14 21:57:33.491773 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.491756 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.495099 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.495074 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 21:57:33.495223 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.495144 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k4kj9\"" Jul 14 21:57:33.496099 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.496082 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 21:57:33.501506 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.501485 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj"] Jul 14 21:57:33.571883 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.571841 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.572091 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.571905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.572091 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.571942 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtpb5\" (UniqueName: \"kubernetes.io/projected/766bf267-dc3d-46ba-8b15-d918114ead9d-kube-api-access-xtpb5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.673244 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.673205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.673411 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.673294 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.673411 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.673331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtpb5\" (UniqueName: \"kubernetes.io/projected/766bf267-dc3d-46ba-8b15-d918114ead9d-kube-api-access-xtpb5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.673633 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.673611 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.673669 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.673635 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.682755 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.682720 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtpb5\" (UniqueName: \"kubernetes.io/projected/766bf267-dc3d-46ba-8b15-d918114ead9d-kube-api-access-xtpb5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.801176 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.801149 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:33.884190 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.884154 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq"] Jul 14 21:57:33.887506 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.887483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:33.901358 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.901331 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq"] Jul 14 21:57:33.946070 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.946047 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj"] Jul 14 21:57:33.948525 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:57:33.948497 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766bf267_dc3d_46ba_8b15_d918114ead9d.slice/crio-afa6fcf97a291683493b8d00231605042183a1274d6461910bad5579ece3866a WatchSource:0}: Error finding container afa6fcf97a291683493b8d00231605042183a1274d6461910bad5579ece3866a: Status 404 returned error can't find the container with id afa6fcf97a291683493b8d00231605042183a1274d6461910bad5579ece3866a Jul 14 21:57:33.976579 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.976551 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:33.976681 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.976587 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:33.976681 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:33.976657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5n4f\" (UniqueName: \"kubernetes.io/projected/fadea2e5-5b30-49d6-974c-8675e4b39316-kube-api-access-h5n4f\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:34.077897 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.077861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:34.078082 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.077905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:34.078082 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.077986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5n4f\" (UniqueName: \"kubernetes.io/projected/fadea2e5-5b30-49d6-974c-8675e4b39316-kube-api-access-h5n4f\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:34.078292 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.078271 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:34.078330 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.078313 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:34.087497 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.087478 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5n4f\" (UniqueName: \"kubernetes.io/projected/fadea2e5-5b30-49d6-974c-8675e4b39316-kube-api-access-h5n4f\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:34.201902 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.201867 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:34.279339 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.279306 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn"] Jul 14 21:57:34.282558 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.282533 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.293739 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.293495 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn"] Jul 14 21:57:34.341667 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.341646 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq"] Jul 14 21:57:34.343746 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:57:34.343717 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadea2e5_5b30_49d6_974c_8675e4b39316.slice/crio-01085dfae4643b54da2ab0f96d51531ef092330679e3f9cfc8e709c3d9d908ae WatchSource:0}: Error finding container 01085dfae4643b54da2ab0f96d51531ef092330679e3f9cfc8e709c3d9d908ae: Status 404 returned error can't find the container with id 01085dfae4643b54da2ab0f96d51531ef092330679e3f9cfc8e709c3d9d908ae Jul 14 21:57:34.380555 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.380523 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.380731 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.380586 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.380731 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.380651 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnjv2\" (UniqueName: \"kubernetes.io/projected/02eb97d0-9f26-4698-87c2-a96b81013779-kube-api-access-rnjv2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.481783 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.481751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.481995 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.481795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.481995 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.481966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnjv2\" (UniqueName: \"kubernetes.io/projected/02eb97d0-9f26-4698-87c2-a96b81013779-kube-api-access-rnjv2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.482258 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.482238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.482345 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.482262 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.491829 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.491797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnjv2\" (UniqueName: \"kubernetes.io/projected/02eb97d0-9f26-4698-87c2-a96b81013779-kube-api-access-rnjv2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.594729 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.594637 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:34.687551 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.686051 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f"] Jul 14 21:57:34.690017 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.689992 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:34.700896 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.700872 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f"] Jul 14 21:57:34.743469 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.743446 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn"] Jul 14 21:57:34.745151 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:57:34.745125 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eb97d0_9f26_4698_87c2_a96b81013779.slice/crio-b12dfec0fdc8a8999eacb6dacbd3508ee86dc2aadb69b305d4d0f626c4438522 WatchSource:0}: Error finding container b12dfec0fdc8a8999eacb6dacbd3508ee86dc2aadb69b305d4d0f626c4438522: Status 404 returned error can't find the container with id b12dfec0fdc8a8999eacb6dacbd3508ee86dc2aadb69b305d4d0f626c4438522 Jul 14 21:57:34.784577 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.784546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:34.784722 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.784646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp7hx\" (UniqueName: \"kubernetes.io/projected/c30f78c7-01aa-4399-af9d-3e30a1da8411-kube-api-access-dp7hx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:34.784722 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.784686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:34.811769 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.811736 2562 generic.go:356] "Generic (PLEG): container finished" podID="766bf267-dc3d-46ba-8b15-d918114ead9d" containerID="50737a3345d572f6fe16b5a4e89403c5718496bb78eee74c70a9ef8c2cbb05de" exitCode=0 Jul 14 21:57:34.811908 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.811812 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" event={"ID":"766bf267-dc3d-46ba-8b15-d918114ead9d","Type":"ContainerDied","Data":"50737a3345d572f6fe16b5a4e89403c5718496bb78eee74c70a9ef8c2cbb05de"} Jul 14 21:57:34.811908 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.811861 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" event={"ID":"766bf267-dc3d-46ba-8b15-d918114ead9d","Type":"ContainerStarted","Data":"afa6fcf97a291683493b8d00231605042183a1274d6461910bad5579ece3866a"} Jul 14 21:57:34.813559 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.813531 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" event={"ID":"02eb97d0-9f26-4698-87c2-a96b81013779","Type":"ContainerStarted","Data":"032fb9b3545aa9b0ee0abddd9e0f30578924dc0d7d5d36c5fb9afce2e6e62b52"} Jul 14 21:57:34.813643 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.813581 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" event={"ID":"02eb97d0-9f26-4698-87c2-a96b81013779","Type":"ContainerStarted","Data":"b12dfec0fdc8a8999eacb6dacbd3508ee86dc2aadb69b305d4d0f626c4438522"} Jul 14 21:57:34.815266 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.815124 2562 generic.go:356] "Generic (PLEG): container finished" podID="fadea2e5-5b30-49d6-974c-8675e4b39316" containerID="89ab2278812a03824f8046bfdf9ad45579e59a18457cd34329b9edd12568e70e" exitCode=0 Jul 14 21:57:34.815266 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.815205 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" event={"ID":"fadea2e5-5b30-49d6-974c-8675e4b39316","Type":"ContainerDied","Data":"89ab2278812a03824f8046bfdf9ad45579e59a18457cd34329b9edd12568e70e"} Jul 14 21:57:34.815266 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.815243 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" event={"ID":"fadea2e5-5b30-49d6-974c-8675e4b39316","Type":"ContainerStarted","Data":"01085dfae4643b54da2ab0f96d51531ef092330679e3f9cfc8e709c3d9d908ae"} Jul 14 21:57:34.885146 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.885110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:34.885332 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.885204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp7hx\" (UniqueName: \"kubernetes.io/projected/c30f78c7-01aa-4399-af9d-3e30a1da8411-kube-api-access-dp7hx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:34.885332 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.885239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:34.885598 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.885574 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:34.885688 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.885638 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:34.902299 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:34.902265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp7hx\" (UniqueName: \"kubernetes.io/projected/c30f78c7-01aa-4399-af9d-3e30a1da8411-kube-api-access-dp7hx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:35.010431 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.010391 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:35.179331 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.179310 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f"] Jul 14 21:57:35.181289 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:57:35.181265 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30f78c7_01aa_4399_af9d_3e30a1da8411.slice/crio-9f5e0dc924780753567f716ca9b7ae344d3893b2942ec80272e375201dd92ffa WatchSource:0}: Error finding container 9f5e0dc924780753567f716ca9b7ae344d3893b2942ec80272e375201dd92ffa: Status 404 returned error can't find the container with id 9f5e0dc924780753567f716ca9b7ae344d3893b2942ec80272e375201dd92ffa Jul 14 21:57:35.822047 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.821945 2562 generic.go:356] "Generic (PLEG): container finished" podID="766bf267-dc3d-46ba-8b15-d918114ead9d" containerID="a235cd8869f8fd046ece0c4db2874255d6b99993d44e6732e83621cab2e8befe" exitCode=0 Jul 14 21:57:35.822426 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.822055 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" event={"ID":"766bf267-dc3d-46ba-8b15-d918114ead9d","Type":"ContainerDied","Data":"a235cd8869f8fd046ece0c4db2874255d6b99993d44e6732e83621cab2e8befe"} Jul 14 21:57:35.823553 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.823534 2562 generic.go:356] "Generic (PLEG): container finished" podID="02eb97d0-9f26-4698-87c2-a96b81013779" containerID="032fb9b3545aa9b0ee0abddd9e0f30578924dc0d7d5d36c5fb9afce2e6e62b52" exitCode=0 Jul 14 21:57:35.823633 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.823615 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" event={"ID":"02eb97d0-9f26-4698-87c2-a96b81013779","Type":"ContainerDied","Data":"032fb9b3545aa9b0ee0abddd9e0f30578924dc0d7d5d36c5fb9afce2e6e62b52"} Jul 14 21:57:35.825435 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.825413 2562 generic.go:356] "Generic (PLEG): container finished" podID="fadea2e5-5b30-49d6-974c-8675e4b39316" containerID="5fede9999faafb22658dc2a444cc0ba699d2e0558325f9b3eb40778c7dc2a4c3" exitCode=0 Jul 14 21:57:35.825546 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.825498 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" event={"ID":"fadea2e5-5b30-49d6-974c-8675e4b39316","Type":"ContainerDied","Data":"5fede9999faafb22658dc2a444cc0ba699d2e0558325f9b3eb40778c7dc2a4c3"} Jul 14 21:57:35.827205 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.827173 2562 generic.go:356] "Generic (PLEG): container finished" podID="c30f78c7-01aa-4399-af9d-3e30a1da8411" containerID="43a76a454206f1905e67620edae0b24bebe851a57c9fb8a90d460e9ca1bbfcaa" exitCode=0 Jul 14 21:57:35.827293 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.827223 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" event={"ID":"c30f78c7-01aa-4399-af9d-3e30a1da8411","Type":"ContainerDied","Data":"43a76a454206f1905e67620edae0b24bebe851a57c9fb8a90d460e9ca1bbfcaa"} Jul 14 21:57:35.827293 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:35.827272 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" event={"ID":"c30f78c7-01aa-4399-af9d-3e30a1da8411","Type":"ContainerStarted","Data":"9f5e0dc924780753567f716ca9b7ae344d3893b2942ec80272e375201dd92ffa"} Jul 14 21:57:36.834177 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:36.834087 2562 generic.go:356] "Generic (PLEG): container finished" podID="02eb97d0-9f26-4698-87c2-a96b81013779" containerID="0e5780fae0932bdaede40be16e2a6b901b4171e0935575f3329446058a928880" exitCode=0 Jul 14 21:57:36.834177 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:36.834150 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" event={"ID":"02eb97d0-9f26-4698-87c2-a96b81013779","Type":"ContainerDied","Data":"0e5780fae0932bdaede40be16e2a6b901b4171e0935575f3329446058a928880"} Jul 14 21:57:36.836444 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:36.836355 2562 generic.go:356] "Generic (PLEG): container finished" podID="fadea2e5-5b30-49d6-974c-8675e4b39316" containerID="00a57d618136afe6e9be485798b53b85c71f3743383353b9d4beebc30dc293b9" exitCode=0 Jul 14 21:57:36.836444 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:36.836390 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" event={"ID":"fadea2e5-5b30-49d6-974c-8675e4b39316","Type":"ContainerDied","Data":"00a57d618136afe6e9be485798b53b85c71f3743383353b9d4beebc30dc293b9"} Jul 14 21:57:36.838988 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:36.838966 2562 generic.go:356] "Generic (PLEG): container finished" podID="766bf267-dc3d-46ba-8b15-d918114ead9d" containerID="93b24c44b36e1cb8b942ecc06ae21a544e626ef1ace7527bcb4aede25028bc27" exitCode=0 Jul 14 21:57:36.839116 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:36.838998 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" event={"ID":"766bf267-dc3d-46ba-8b15-d918114ead9d","Type":"ContainerDied","Data":"93b24c44b36e1cb8b942ecc06ae21a544e626ef1ace7527bcb4aede25028bc27"} Jul 14 21:57:37.845625 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:37.845581 2562 generic.go:356] "Generic (PLEG): container finished" podID="02eb97d0-9f26-4698-87c2-a96b81013779" containerID="f12a39425bf116ff8289fd1a4ab95d3201471fbf786b1ff75530589aea0ec1a6" exitCode=0 Jul 14 21:57:37.846078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:37.845657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" event={"ID":"02eb97d0-9f26-4698-87c2-a96b81013779","Type":"ContainerDied","Data":"f12a39425bf116ff8289fd1a4ab95d3201471fbf786b1ff75530589aea0ec1a6"} Jul 14 21:57:37.847584 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:37.847561 2562 generic.go:356] "Generic (PLEG): container finished" podID="c30f78c7-01aa-4399-af9d-3e30a1da8411" containerID="b9ec403af9f6b4050f472509957c52b6269b025f8705680bc49c4560377e6de8" exitCode=0 Jul 14 21:57:37.847704 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:37.847587 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" event={"ID":"c30f78c7-01aa-4399-af9d-3e30a1da8411","Type":"ContainerDied","Data":"b9ec403af9f6b4050f472509957c52b6269b025f8705680bc49c4560377e6de8"} Jul 14 21:57:37.985430 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:37.985404 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:38.014084 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.014049 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-util\") pod \"766bf267-dc3d-46ba-8b15-d918114ead9d\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " Jul 14 21:57:38.014202 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.014125 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-bundle\") pod \"766bf267-dc3d-46ba-8b15-d918114ead9d\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " Jul 14 21:57:38.014202 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.014196 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtpb5\" (UniqueName: \"kubernetes.io/projected/766bf267-dc3d-46ba-8b15-d918114ead9d-kube-api-access-xtpb5\") pod \"766bf267-dc3d-46ba-8b15-d918114ead9d\" (UID: \"766bf267-dc3d-46ba-8b15-d918114ead9d\") " Jul 14 21:57:38.014991 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.014897 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-bundle" (OuterVolumeSpecName: "bundle") pod "766bf267-dc3d-46ba-8b15-d918114ead9d" (UID: "766bf267-dc3d-46ba-8b15-d918114ead9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:57:38.016716 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.016690 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766bf267-dc3d-46ba-8b15-d918114ead9d-kube-api-access-xtpb5" (OuterVolumeSpecName: "kube-api-access-xtpb5") pod "766bf267-dc3d-46ba-8b15-d918114ead9d" (UID: "766bf267-dc3d-46ba-8b15-d918114ead9d"). InnerVolumeSpecName "kube-api-access-xtpb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:57:38.020518 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.020491 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-util" (OuterVolumeSpecName: "util") pod "766bf267-dc3d-46ba-8b15-d918114ead9d" (UID: "766bf267-dc3d-46ba-8b15-d918114ead9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:57:38.060476 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.060440 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:38.115823 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.115738 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5n4f\" (UniqueName: \"kubernetes.io/projected/fadea2e5-5b30-49d6-974c-8675e4b39316-kube-api-access-h5n4f\") pod \"fadea2e5-5b30-49d6-974c-8675e4b39316\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " Jul 14 21:57:38.115823 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.115789 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-bundle\") pod \"fadea2e5-5b30-49d6-974c-8675e4b39316\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " Jul 14 21:57:38.115823 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.115823 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-util\") pod \"fadea2e5-5b30-49d6-974c-8675e4b39316\" (UID: \"fadea2e5-5b30-49d6-974c-8675e4b39316\") " Jul 14 21:57:38.116127 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.116014 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-util\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:38.116127 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.116050 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/766bf267-dc3d-46ba-8b15-d918114ead9d-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:38.116127 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.116061 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtpb5\" (UniqueName: \"kubernetes.io/projected/766bf267-dc3d-46ba-8b15-d918114ead9d-kube-api-access-xtpb5\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:38.116396 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.116359 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-bundle" (OuterVolumeSpecName: "bundle") pod "fadea2e5-5b30-49d6-974c-8675e4b39316" (UID: "fadea2e5-5b30-49d6-974c-8675e4b39316"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:57:38.117804 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.117778 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadea2e5-5b30-49d6-974c-8675e4b39316-kube-api-access-h5n4f" (OuterVolumeSpecName: "kube-api-access-h5n4f") pod "fadea2e5-5b30-49d6-974c-8675e4b39316" (UID: "fadea2e5-5b30-49d6-974c-8675e4b39316"). InnerVolumeSpecName "kube-api-access-h5n4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:57:38.121543 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.121523 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-util" (OuterVolumeSpecName: "util") pod "fadea2e5-5b30-49d6-974c-8675e4b39316" (UID: "fadea2e5-5b30-49d6-974c-8675e4b39316"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:57:38.217452 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.217419 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-util\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:38.217452 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.217449 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5n4f\" (UniqueName: \"kubernetes.io/projected/fadea2e5-5b30-49d6-974c-8675e4b39316-kube-api-access-h5n4f\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:38.217452 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.217458 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fadea2e5-5b30-49d6-974c-8675e4b39316-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:38.854935 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.854906 2562 generic.go:356] "Generic (PLEG): container finished" podID="c30f78c7-01aa-4399-af9d-3e30a1da8411" containerID="6de1aa903a4951ab4a994d291acea48f6d051db26f64aebd8d545ff365fa1890" exitCode=0 Jul 14 21:57:38.855318 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.854993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" event={"ID":"c30f78c7-01aa-4399-af9d-3e30a1da8411","Type":"ContainerDied","Data":"6de1aa903a4951ab4a994d291acea48f6d051db26f64aebd8d545ff365fa1890"} Jul 14 21:57:38.856747 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.856730 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" Jul 14 21:57:38.856857 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.856757 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj" event={"ID":"766bf267-dc3d-46ba-8b15-d918114ead9d","Type":"ContainerDied","Data":"afa6fcf97a291683493b8d00231605042183a1274d6461910bad5579ece3866a"} Jul 14 21:57:38.856857 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.856801 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afa6fcf97a291683493b8d00231605042183a1274d6461910bad5579ece3866a" Jul 14 21:57:38.858915 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.858644 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" Jul 14 21:57:38.858915 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.858656 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq" event={"ID":"fadea2e5-5b30-49d6-974c-8675e4b39316","Type":"ContainerDied","Data":"01085dfae4643b54da2ab0f96d51531ef092330679e3f9cfc8e709c3d9d908ae"} Jul 14 21:57:38.858915 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.858726 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01085dfae4643b54da2ab0f96d51531ef092330679e3f9cfc8e709c3d9d908ae" Jul 14 21:57:38.990258 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:38.990229 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:39.024434 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.024404 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-bundle\") pod \"02eb97d0-9f26-4698-87c2-a96b81013779\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " Jul 14 21:57:39.024621 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.024473 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnjv2\" (UniqueName: \"kubernetes.io/projected/02eb97d0-9f26-4698-87c2-a96b81013779-kube-api-access-rnjv2\") pod \"02eb97d0-9f26-4698-87c2-a96b81013779\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " Jul 14 21:57:39.024621 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.024559 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-util\") pod \"02eb97d0-9f26-4698-87c2-a96b81013779\" (UID: \"02eb97d0-9f26-4698-87c2-a96b81013779\") " Jul 14 21:57:39.025121 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.025088 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-bundle" (OuterVolumeSpecName: "bundle") pod "02eb97d0-9f26-4698-87c2-a96b81013779" (UID: "02eb97d0-9f26-4698-87c2-a96b81013779"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:57:39.026560 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.026536 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eb97d0-9f26-4698-87c2-a96b81013779-kube-api-access-rnjv2" (OuterVolumeSpecName: "kube-api-access-rnjv2") pod "02eb97d0-9f26-4698-87c2-a96b81013779" (UID: "02eb97d0-9f26-4698-87c2-a96b81013779"). InnerVolumeSpecName "kube-api-access-rnjv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:57:39.030312 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.030284 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-util" (OuterVolumeSpecName: "util") pod "02eb97d0-9f26-4698-87c2-a96b81013779" (UID: "02eb97d0-9f26-4698-87c2-a96b81013779"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:57:39.126238 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.126160 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnjv2\" (UniqueName: \"kubernetes.io/projected/02eb97d0-9f26-4698-87c2-a96b81013779-kube-api-access-rnjv2\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:39.126238 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.126185 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-util\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:39.126238 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.126197 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02eb97d0-9f26-4698-87c2-a96b81013779-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:39.865300 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.865273 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" Jul 14 21:57:39.865703 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.865304 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn" event={"ID":"02eb97d0-9f26-4698-87c2-a96b81013779","Type":"ContainerDied","Data":"b12dfec0fdc8a8999eacb6dacbd3508ee86dc2aadb69b305d4d0f626c4438522"} Jul 14 21:57:39.865703 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:39.865344 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b12dfec0fdc8a8999eacb6dacbd3508ee86dc2aadb69b305d4d0f626c4438522" Jul 14 21:57:40.000178 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.000153 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:40.035232 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.035204 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-bundle\") pod \"c30f78c7-01aa-4399-af9d-3e30a1da8411\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " Jul 14 21:57:40.035421 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.035339 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-util\") pod \"c30f78c7-01aa-4399-af9d-3e30a1da8411\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " Jul 14 21:57:40.035421 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.035379 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp7hx\" (UniqueName: \"kubernetes.io/projected/c30f78c7-01aa-4399-af9d-3e30a1da8411-kube-api-access-dp7hx\") pod \"c30f78c7-01aa-4399-af9d-3e30a1da8411\" (UID: \"c30f78c7-01aa-4399-af9d-3e30a1da8411\") " Jul 14 21:57:40.035857 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.035831 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-bundle" (OuterVolumeSpecName: "bundle") pod "c30f78c7-01aa-4399-af9d-3e30a1da8411" (UID: "c30f78c7-01aa-4399-af9d-3e30a1da8411"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:57:40.037721 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.037698 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30f78c7-01aa-4399-af9d-3e30a1da8411-kube-api-access-dp7hx" (OuterVolumeSpecName: "kube-api-access-dp7hx") pod "c30f78c7-01aa-4399-af9d-3e30a1da8411" (UID: "c30f78c7-01aa-4399-af9d-3e30a1da8411"). InnerVolumeSpecName "kube-api-access-dp7hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:57:40.044516 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.044464 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-util" (OuterVolumeSpecName: "util") pod "c30f78c7-01aa-4399-af9d-3e30a1da8411" (UID: "c30f78c7-01aa-4399-af9d-3e30a1da8411"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:57:40.136417 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.136317 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-util\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:40.136417 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.136349 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp7hx\" (UniqueName: \"kubernetes.io/projected/c30f78c7-01aa-4399-af9d-3e30a1da8411-kube-api-access-dp7hx\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:40.136417 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.136358 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c30f78c7-01aa-4399-af9d-3e30a1da8411-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:57:40.872093 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.872056 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" event={"ID":"c30f78c7-01aa-4399-af9d-3e30a1da8411","Type":"ContainerDied","Data":"9f5e0dc924780753567f716ca9b7ae344d3893b2942ec80272e375201dd92ffa"} Jul 14 21:57:40.872093 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.872089 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f" Jul 14 21:57:40.872611 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:57:40.872109 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f5e0dc924780753567f716ca9b7ae344d3893b2942ec80272e375201dd92ffa" Jul 14 21:58:01.372301 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372270 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d8744fcb7-fr6gx"] Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372607 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30f78c7-01aa-4399-af9d-3e30a1da8411" containerName="extract" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372618 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30f78c7-01aa-4399-af9d-3e30a1da8411" containerName="extract" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372629 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30f78c7-01aa-4399-af9d-3e30a1da8411" containerName="util" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372636 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30f78c7-01aa-4399-af9d-3e30a1da8411" containerName="util" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372643 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fadea2e5-5b30-49d6-974c-8675e4b39316" containerName="extract" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372649 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fadea2e5-5b30-49d6-974c-8675e4b39316" containerName="extract" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372655 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02eb97d0-9f26-4698-87c2-a96b81013779" containerName="util" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372660 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02eb97d0-9f26-4698-87c2-a96b81013779" containerName="util" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372667 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02eb97d0-9f26-4698-87c2-a96b81013779" containerName="extract" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372671 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02eb97d0-9f26-4698-87c2-a96b81013779" containerName="extract" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372679 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fadea2e5-5b30-49d6-974c-8675e4b39316" containerName="pull" Jul 14 21:58:01.372682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372684 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fadea2e5-5b30-49d6-974c-8675e4b39316" containerName="pull" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372691 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766bf267-dc3d-46ba-8b15-d918114ead9d" containerName="pull" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372698 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="766bf267-dc3d-46ba-8b15-d918114ead9d" containerName="pull" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372707 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fadea2e5-5b30-49d6-974c-8675e4b39316" containerName="util" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372712 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fadea2e5-5b30-49d6-974c-8675e4b39316" containerName="util" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372718 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766bf267-dc3d-46ba-8b15-d918114ead9d" containerName="extract" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372722 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="766bf267-dc3d-46ba-8b15-d918114ead9d" containerName="extract" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372730 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766bf267-dc3d-46ba-8b15-d918114ead9d" containerName="util" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372735 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="766bf267-dc3d-46ba-8b15-d918114ead9d" containerName="util" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372740 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30f78c7-01aa-4399-af9d-3e30a1da8411" containerName="pull" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372744 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30f78c7-01aa-4399-af9d-3e30a1da8411" containerName="pull" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372750 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02eb97d0-9f26-4698-87c2-a96b81013779" containerName="pull" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372754 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02eb97d0-9f26-4698-87c2-a96b81013779" containerName="pull" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372800 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fadea2e5-5b30-49d6-974c-8675e4b39316" containerName="extract" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372807 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="766bf267-dc3d-46ba-8b15-d918114ead9d" containerName="extract" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372815 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30f78c7-01aa-4399-af9d-3e30a1da8411" containerName="extract" Jul 14 21:58:01.373078 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.372820 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02eb97d0-9f26-4698-87c2-a96b81013779" containerName="extract" Jul 14 21:58:01.375578 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.375562 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.401306 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.401281 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d8744fcb7-fr6gx"] Jul 14 21:58:01.524641 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.524612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-trusted-ca-bundle\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.524823 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.524649 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-console-config\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.524823 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.524667 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-service-ca\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.524823 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.524686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-console-serving-cert\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.524823 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.524759 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-console-oauth-config\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.524823 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.524821 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hsd7\" (UniqueName: \"kubernetes.io/projected/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-kube-api-access-7hsd7\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.525088 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.524900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-oauth-serving-cert\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.626262 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.626166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-trusted-ca-bundle\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.626262 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.626211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-console-config\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.626262 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.626228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-service-ca\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.626262 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.626247 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-console-serving-cert\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.626612 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.626365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-console-oauth-config\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.626612 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.626430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hsd7\" (UniqueName: \"kubernetes.io/projected/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-kube-api-access-7hsd7\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.626612 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.626470 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-oauth-serving-cert\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.627114 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.627086 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-console-config\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.627231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.627087 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-service-ca\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.627231 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.627154 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-oauth-serving-cert\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.627407 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.627387 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-trusted-ca-bundle\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.628941 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.628914 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-console-oauth-config\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.629016 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.628991 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-console-serving-cert\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.646950 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.646918 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hsd7\" (UniqueName: \"kubernetes.io/projected/4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b-kube-api-access-7hsd7\") pod \"console-d8744fcb7-fr6gx\" (UID: \"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b\") " pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.685169 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.685137 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:01.851457 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.851434 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d8744fcb7-fr6gx"] Jul 14 21:58:01.853102 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:58:01.853077 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfa8389_30cd_4bc5_aaa5_65acd9e75b2b.slice/crio-0d54b1b59e2071bd4060e4947574f31279b44c4a1bbd2d0570d6260c599b962f WatchSource:0}: Error finding container 0d54b1b59e2071bd4060e4947574f31279b44c4a1bbd2d0570d6260c599b962f: Status 404 returned error can't find the container with id 0d54b1b59e2071bd4060e4947574f31279b44c4a1bbd2d0570d6260c599b962f Jul 14 21:58:01.979456 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.979423 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8744fcb7-fr6gx" event={"ID":"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b","Type":"ContainerStarted","Data":"af617f812072f84a68f3f22a6dfa0fe0c433091992c774af04cbd725ff1f0fa8"} Jul 14 21:58:01.979619 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:01.979476 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8744fcb7-fr6gx" event={"ID":"4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b","Type":"ContainerStarted","Data":"0d54b1b59e2071bd4060e4947574f31279b44c4a1bbd2d0570d6260c599b962f"} Jul 14 21:58:02.037447 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:02.037394 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d8744fcb7-fr6gx" podStartSLOduration=1.037378069 podStartE2EDuration="1.037378069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:58:02.035557816 +0000 UTC m=+416.885027898" watchObservedRunningTime="2026-07-14 21:58:02.037378069 +0000 UTC m=+416.886848109" Jul 14 21:58:11.685445 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:11.685409 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:11.685918 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:11.685470 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:11.690065 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:11.690043 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:12.034544 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:12.034517 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d8744fcb7-fr6gx" Jul 14 21:58:12.158485 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:12.158452 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6679c7c685-zmvmj"] Jul 14 21:58:37.178371 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.178256 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6679c7c685-zmvmj" podUID="85a80b20-95a0-455c-b391-5dab2112d6cd" containerName="console" containerID="cri-o://3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce" gracePeriod=15 Jul 14 21:58:37.431064 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.430989 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:58:37.545492 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.545455 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-oauth-config\") pod \"85a80b20-95a0-455c-b391-5dab2112d6cd\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " Jul 14 21:58:37.545718 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.545582 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-trusted-ca-bundle\") pod \"85a80b20-95a0-455c-b391-5dab2112d6cd\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " Jul 14 21:58:37.545718 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.545631 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-service-ca\") pod \"85a80b20-95a0-455c-b391-5dab2112d6cd\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " Jul 14 21:58:37.545718 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.545654 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-oauth-serving-cert\") pod \"85a80b20-95a0-455c-b391-5dab2112d6cd\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " Jul 14 21:58:37.545718 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.545686 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssm6n\" (UniqueName: \"kubernetes.io/projected/85a80b20-95a0-455c-b391-5dab2112d6cd-kube-api-access-ssm6n\") pod \"85a80b20-95a0-455c-b391-5dab2112d6cd\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " Jul 14 21:58:37.546066 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.545730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-console-config\") pod \"85a80b20-95a0-455c-b391-5dab2112d6cd\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " Jul 14 21:58:37.546066 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.545757 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-serving-cert\") pod \"85a80b20-95a0-455c-b391-5dab2112d6cd\" (UID: \"85a80b20-95a0-455c-b391-5dab2112d6cd\") " Jul 14 21:58:37.546189 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.546136 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-service-ca" (OuterVolumeSpecName: "service-ca") pod "85a80b20-95a0-455c-b391-5dab2112d6cd" (UID: "85a80b20-95a0-455c-b391-5dab2112d6cd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:58:37.546189 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.546147 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "85a80b20-95a0-455c-b391-5dab2112d6cd" (UID: "85a80b20-95a0-455c-b391-5dab2112d6cd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:58:37.546287 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.546176 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-console-config" (OuterVolumeSpecName: "console-config") pod "85a80b20-95a0-455c-b391-5dab2112d6cd" (UID: "85a80b20-95a0-455c-b391-5dab2112d6cd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:58:37.546561 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.546532 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "85a80b20-95a0-455c-b391-5dab2112d6cd" (UID: "85a80b20-95a0-455c-b391-5dab2112d6cd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:58:37.548050 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.547991 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "85a80b20-95a0-455c-b391-5dab2112d6cd" (UID: "85a80b20-95a0-455c-b391-5dab2112d6cd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:58:37.548155 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.548070 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "85a80b20-95a0-455c-b391-5dab2112d6cd" (UID: "85a80b20-95a0-455c-b391-5dab2112d6cd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:58:37.548155 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.548113 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a80b20-95a0-455c-b391-5dab2112d6cd-kube-api-access-ssm6n" (OuterVolumeSpecName: "kube-api-access-ssm6n") pod "85a80b20-95a0-455c-b391-5dab2112d6cd" (UID: "85a80b20-95a0-455c-b391-5dab2112d6cd"). InnerVolumeSpecName "kube-api-access-ssm6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:58:37.647247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.647212 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-trusted-ca-bundle\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:58:37.647247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.647240 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:58:37.647247 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.647249 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-oauth-serving-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:58:37.647475 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.647258 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssm6n\" (UniqueName: \"kubernetes.io/projected/85a80b20-95a0-455c-b391-5dab2112d6cd-kube-api-access-ssm6n\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:58:37.647475 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.647267 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85a80b20-95a0-455c-b391-5dab2112d6cd-console-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:58:37.647475 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.647275 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-serving-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:58:37.647475 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:37.647284 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85a80b20-95a0-455c-b391-5dab2112d6cd-console-oauth-config\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 21:58:38.166552 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:38.166519 2562 generic.go:356] "Generic (PLEG): container finished" podID="85a80b20-95a0-455c-b391-5dab2112d6cd" containerID="3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce" exitCode=0 Jul 14 21:58:38.166726 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:38.166570 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6679c7c685-zmvmj" Jul 14 21:58:38.166726 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:38.166597 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6679c7c685-zmvmj" event={"ID":"85a80b20-95a0-455c-b391-5dab2112d6cd","Type":"ContainerDied","Data":"3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce"} Jul 14 21:58:38.166726 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:38.166669 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6679c7c685-zmvmj" event={"ID":"85a80b20-95a0-455c-b391-5dab2112d6cd","Type":"ContainerDied","Data":"c63905c2beb0cd115a3f723d9a5898b02cf9b6cb7a1319ee61066fbbc3b05b6f"} Jul 14 21:58:38.166726 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:38.166689 2562 scope.go:117] "RemoveContainer" containerID="3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce" Jul 14 21:58:38.175821 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:38.175807 2562 scope.go:117] "RemoveContainer" containerID="3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce" Jul 14 21:58:38.176072 ip-10-0-141-219 kubenswrapper[2562]: E0714 21:58:38.176055 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce\": container with ID starting with 3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce not found: ID does not exist" containerID="3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce" Jul 14 21:58:38.176116 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:38.176081 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce"} err="failed to get container status \"3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce\": rpc error: code = NotFound desc = could not find container \"3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce\": container with ID starting with 3ab589dcd95e28926b6f4ad4bde7c8fa4f37508ce1bf188482d1f88818fac2ce not found: ID does not exist" Jul 14 21:58:38.188100 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:38.188076 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6679c7c685-zmvmj"] Jul 14 21:58:38.194012 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:38.193980 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6679c7c685-zmvmj"] Jul 14 21:58:38.810136 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:58:38.810103 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85a80b20-95a0-455c-b391-5dab2112d6cd" path="/var/lib/kubelet/pods/85a80b20-95a0-455c-b391-5dab2112d6cd/volumes" Jul 14 21:59:03.428146 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.428101 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9"] Jul 14 21:59:03.428650 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.428631 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a80b20-95a0-455c-b391-5dab2112d6cd" containerName="console" Jul 14 21:59:03.428694 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.428655 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a80b20-95a0-455c-b391-5dab2112d6cd" containerName="console" Jul 14 21:59:03.428769 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.428760 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a80b20-95a0-455c-b391-5dab2112d6cd" containerName="console" Jul 14 21:59:03.431706 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.431690 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.434353 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.434335 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8lw4r\"" Jul 14 21:59:03.442440 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.442416 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9"] Jul 14 21:59:03.568949 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.568911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.568949 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.568949 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.569191 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.568972 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.569191 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.568996 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.569191 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.569066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.569191 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.569104 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.569191 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.569138 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.569191 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.569175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbx5f\" (UniqueName: \"kubernetes.io/projected/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-kube-api-access-kbx5f\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.569659 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.569203 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670040 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670002 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670229 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670229 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbx5f\" (UniqueName: \"kubernetes.io/projected/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-kube-api-access-kbx5f\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670229 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670229 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670162 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670229 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670229 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670552 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670552 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670552 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670540 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670682 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670664 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670749 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670679 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.670916 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.670892 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.672521 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.672495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.672612 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.672527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.681135 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.681067 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.681837 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.681814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbx5f\" (UniqueName: \"kubernetes.io/projected/494a30d6-6cb0-4f37-9f22-753ac3f31fb5-kube-api-access-kbx5f\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vqgb9\" (UID: \"494a30d6-6cb0-4f37-9f22-753ac3f31fb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.742732 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.742697 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:03.883800 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.883774 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9"] Jul 14 21:59:03.885400 ip-10-0-141-219 kubenswrapper[2562]: W0714 21:59:03.885369 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494a30d6_6cb0_4f37_9f22_753ac3f31fb5.slice/crio-8ad81c06a7576f9351e9280984191fd33de54ba0b582633843ab9973934b14b3 WatchSource:0}: Error finding container 8ad81c06a7576f9351e9280984191fd33de54ba0b582633843ab9973934b14b3: Status 404 returned error can't find the container with id 8ad81c06a7576f9351e9280984191fd33de54ba0b582633843ab9973934b14b3 Jul 14 21:59:03.887465 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.887430 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 21:59:03.887542 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.887501 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 21:59:03.887542 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:03.887530 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 21:59:04.300512 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:04.300474 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" event={"ID":"494a30d6-6cb0-4f37-9f22-753ac3f31fb5","Type":"ContainerStarted","Data":"007d70a705e8353cae98736dfc059e21096d3ed0adb1efaf61da27db8066a83e"} Jul 14 21:59:04.300690 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:04.300536 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" event={"ID":"494a30d6-6cb0-4f37-9f22-753ac3f31fb5","Type":"ContainerStarted","Data":"8ad81c06a7576f9351e9280984191fd33de54ba0b582633843ab9973934b14b3"} Jul 14 21:59:04.321877 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:04.321830 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" podStartSLOduration=1.321813732 podStartE2EDuration="1.321813732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:59:04.319741198 +0000 UTC m=+479.169211255" watchObservedRunningTime="2026-07-14 21:59:04.321813732 +0000 UTC m=+479.171283770" Jul 14 21:59:04.743738 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:04.743642 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:04.748709 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:04.748685 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:05.305772 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:05.305737 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 21:59:05.306719 ip-10-0-141-219 kubenswrapper[2562]: I0714 21:59:05.306695 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 22:00:23.298748 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.298663 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-kfknj"] Jul 14 22:00:23.302000 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.301983 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" Jul 14 22:00:23.304993 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.304974 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 14 22:00:23.305149 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.305094 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mwv4h\"" Jul 14 22:00:23.313828 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.313806 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-kfknj"] Jul 14 22:00:23.358219 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.358193 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8397de-5442-4164-8820-24b34ba09a0c-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-kfknj\" (UID: \"1c8397de-5442-4164-8820-24b34ba09a0c\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" Jul 14 22:00:23.358377 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.358223 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldrt\" (UniqueName: \"kubernetes.io/projected/1c8397de-5442-4164-8820-24b34ba09a0c-kube-api-access-6ldrt\") pod \"payload-processing-f9b8d9b9b-kfknj\" (UID: \"1c8397de-5442-4164-8820-24b34ba09a0c\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" Jul 14 22:00:23.459664 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.459629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8397de-5442-4164-8820-24b34ba09a0c-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-kfknj\" (UID: \"1c8397de-5442-4164-8820-24b34ba09a0c\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" Jul 14 22:00:23.459836 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.459676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldrt\" (UniqueName: \"kubernetes.io/projected/1c8397de-5442-4164-8820-24b34ba09a0c-kube-api-access-6ldrt\") pod \"payload-processing-f9b8d9b9b-kfknj\" (UID: \"1c8397de-5442-4164-8820-24b34ba09a0c\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" Jul 14 22:00:23.460423 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.460384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8397de-5442-4164-8820-24b34ba09a0c-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-kfknj\" (UID: \"1c8397de-5442-4164-8820-24b34ba09a0c\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" Jul 14 22:00:23.472596 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.472565 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldrt\" (UniqueName: \"kubernetes.io/projected/1c8397de-5442-4164-8820-24b34ba09a0c-kube-api-access-6ldrt\") pod \"payload-processing-f9b8d9b9b-kfknj\" (UID: \"1c8397de-5442-4164-8820-24b34ba09a0c\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" Jul 14 22:00:23.613215 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.613118 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" Jul 14 22:00:23.745424 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:23.745399 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-kfknj"] Jul 14 22:00:23.746454 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:00:23.746431 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8397de_5442_4164_8820_24b34ba09a0c.slice/crio-17dbe11555bac6f05188ce7aed899bd0222f7ce54fc85591793836ec95932caa WatchSource:0}: Error finding container 17dbe11555bac6f05188ce7aed899bd0222f7ce54fc85591793836ec95932caa: Status 404 returned error can't find the container with id 17dbe11555bac6f05188ce7aed899bd0222f7ce54fc85591793836ec95932caa Jul 14 22:00:24.719327 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:24.719277 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" event={"ID":"1c8397de-5442-4164-8820-24b34ba09a0c","Type":"ContainerStarted","Data":"17dbe11555bac6f05188ce7aed899bd0222f7ce54fc85591793836ec95932caa"} Jul 14 22:00:25.726259 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:25.726163 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" event={"ID":"1c8397de-5442-4164-8820-24b34ba09a0c","Type":"ContainerStarted","Data":"8aa4b490e7d52dc7b8f896dde99f4e03a3f468a9f48ec808f3c71c002d7c37c7"} Jul 14 22:00:25.726637 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:25.726298 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" Jul 14 22:00:25.743498 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:25.743445 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" podStartSLOduration=1.217737276 podStartE2EDuration="2.743429808s" totalImagesPullingTime="1.525692532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:00:25.741407285 +0000 UTC m=+560.590877325" watchObservedRunningTime="2026-07-14 22:00:25.743429808 +0000 UTC m=+560.592899847" Jul 14 22:00:31.765174 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:31.765068 2562 generic.go:356] "Generic (PLEG): container finished" podID="494a30d6-6cb0-4f37-9f22-753ac3f31fb5" containerID="007d70a705e8353cae98736dfc059e21096d3ed0adb1efaf61da27db8066a83e" exitCode=0 Jul 14 22:00:31.765174 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:31.765142 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" event={"ID":"494a30d6-6cb0-4f37-9f22-753ac3f31fb5","Type":"ContainerDied","Data":"007d70a705e8353cae98736dfc059e21096d3ed0adb1efaf61da27db8066a83e"} Jul 14 22:00:31.765605 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:31.765517 2562 scope.go:117] "RemoveContainer" containerID="007d70a705e8353cae98736dfc059e21096d3ed0adb1efaf61da27db8066a83e" Jul 14 22:00:31.766516 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:31.766490 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:00:31.766584 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:31.766558 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:00:31.766621 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:31.766591 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:00:32.771850 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:32.771812 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" event={"ID":"494a30d6-6cb0-4f37-9f22-753ac3f31fb5","Type":"ContainerStarted","Data":"8201240f3ecab7ca289188bfc4c311f997b93a14550b4320037ad2c52c69189d"} Jul 14 22:00:33.743689 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:33.743660 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 22:00:33.744709 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:33.744686 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-vqgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 22:00:33.744777 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:33.744735 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" podUID="494a30d6-6cb0-4f37-9f22-753ac3f31fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 22:00:34.743679 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:34.743639 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-vqgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 22:00:34.744107 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:34.743695 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" podUID="494a30d6-6cb0-4f37-9f22-753ac3f31fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 22:00:35.307081 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:35.306359 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 22:00:35.743398 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:35.743304 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-vqgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 22:00:35.743398 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:35.743361 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" podUID="494a30d6-6cb0-4f37-9f22-753ac3f31fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 22:00:36.733588 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:36.733554 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-kfknj" Jul 14 22:00:36.743558 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:36.743528 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-vqgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 22:00:36.743715 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:36.743576 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" podUID="494a30d6-6cb0-4f37-9f22-753ac3f31fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 22:00:37.747662 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:37.747627 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 22:00:37.748545 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:00:37.748527 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vqgb9" Jul 14 22:15:00.155453 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:00.155367 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z"] Jul 14 22:15:00.158736 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:00.158717 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z" Jul 14 22:15:00.161662 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:00.161638 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:15:00.168400 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:00.168379 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z" Jul 14 22:15:00.194315 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:00.194275 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z"] Jul 14 22:15:00.309893 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:00.309871 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z"] Jul 14 22:15:00.312006 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:15:00.311974 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42c7458_2bb7_4586_8ee4_ef05755b7db9.slice/crio-03019ae00276afe50b4a36e1801b63ff6d35dc9d9d0bf6d1b2ba47ef9e5d3248 WatchSource:0}: Error finding container 03019ae00276afe50b4a36e1801b63ff6d35dc9d9d0bf6d1b2ba47ef9e5d3248: Status 404 returned error can't find the container with id 03019ae00276afe50b4a36e1801b63ff6d35dc9d9d0bf6d1b2ba47ef9e5d3248 Jul 14 22:15:00.313802 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:00.313784 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:15:01.247194 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:01.247157 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z" event={"ID":"c42c7458-2bb7-4586-8ee4-ef05755b7db9","Type":"ContainerStarted","Data":"03019ae00276afe50b4a36e1801b63ff6d35dc9d9d0bf6d1b2ba47ef9e5d3248"} Jul 14 22:15:03.266597 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:03.266563 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z" event={"ID":"c42c7458-2bb7-4586-8ee4-ef05755b7db9","Type":"ContainerStarted","Data":"f2630f80b374d4920c0f6360a1bec22b57225f71ca1e6f1a772f44f6fd6e71cf"} Jul 14 22:15:03.282363 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:03.282318 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z" podStartSLOduration=0.929775229 podStartE2EDuration="3.282303214s" totalImagesPullingTime="2.352527985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:15:03.281118544 +0000 UTC m=+1438.130588585" watchObservedRunningTime="2026-07-14 22:15:03.282303214 +0000 UTC m=+1438.131773262" Jul 14 22:15:04.273728 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:04.273693 2562 generic.go:356] "Generic (PLEG): container finished" podID="c42c7458-2bb7-4586-8ee4-ef05755b7db9" containerID="f2630f80b374d4920c0f6360a1bec22b57225f71ca1e6f1a772f44f6fd6e71cf" exitCode=0 Jul 14 22:15:04.274172 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:04.273781 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z" event={"ID":"c42c7458-2bb7-4586-8ee4-ef05755b7db9","Type":"ContainerDied","Data":"f2630f80b374d4920c0f6360a1bec22b57225f71ca1e6f1a772f44f6fd6e71cf"} Jul 14 22:15:05.412933 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:05.412904 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z" Jul 14 22:15:06.285807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:06.285781 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z" Jul 14 22:15:06.285807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:06.285804 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734455-cm55z" event={"ID":"c42c7458-2bb7-4586-8ee4-ef05755b7db9","Type":"ContainerDied","Data":"03019ae00276afe50b4a36e1801b63ff6d35dc9d9d0bf6d1b2ba47ef9e5d3248"} Jul 14 22:15:06.286226 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:15:06.285844 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03019ae00276afe50b4a36e1801b63ff6d35dc9d9d0bf6d1b2ba47ef9e5d3248" Jul 14 22:16:59.806994 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.806963 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf"] Jul 14 22:16:59.807510 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.807332 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42c7458-2bb7-4586-8ee4-ef05755b7db9" containerName="cleanup" Jul 14 22:16:59.807510 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.807350 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42c7458-2bb7-4586-8ee4-ef05755b7db9" containerName="cleanup" Jul 14 22:16:59.807510 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.807415 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42c7458-2bb7-4586-8ee4-ef05755b7db9" containerName="cleanup" Jul 14 22:16:59.810701 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.810680 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.813781 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.813755 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5574436c-openshift-default-dockercfg-5t278\"" Jul 14 22:16:59.829071 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.829035 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf"] Jul 14 22:16:59.897542 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.897501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-podinfo\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.897708 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.897558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-token\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.897708 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.897584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-socket\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.897708 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.897627 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-certs\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.897708 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.897649 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-envoy\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.897708 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.897675 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istiod-ca-cert\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.897878 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.897727 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-data\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.897878 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.897755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-credential-socket\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.897878 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.897776 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzthm\" (UniqueName: \"kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-kube-api-access-mzthm\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.998932 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.998896 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-data\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.998932 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.998935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-credential-socket\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999225 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.998953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzthm\" (UniqueName: \"kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-kube-api-access-mzthm\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999225 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999012 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-podinfo\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999225 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-token\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999225 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-socket\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999225 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-certs\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999225 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-envoy\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999541 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istiod-ca-cert\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999541 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999372 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-data\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999541 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999419 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-credential-socket\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999541 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999528 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-socket\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999724 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999570 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-certs\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:16:59.999824 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:16:59.999807 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istiod-ca-cert\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:00.001505 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.001484 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-envoy\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:00.001729 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.001709 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-podinfo\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:00.007507 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.007477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzthm\" (UniqueName: \"kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-kube-api-access-mzthm\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:00.007624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.007521 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-token\") pod \"e2e-ait-5574436c-openshift-default-6d6695646f-whqrf\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:00.122916 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.122810 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:00.267075 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.267047 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf"] Jul 14 22:17:00.269549 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:17:00.269521 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04248810_5eb4_4aba_8ce3_006dc32f0b8c.slice/crio-8175bb00a946b25e37342a0256cc75bd99d5392289082319f2378b6bed20c13c WatchSource:0}: Error finding container 8175bb00a946b25e37342a0256cc75bd99d5392289082319f2378b6bed20c13c: Status 404 returned error can't find the container with id 8175bb00a946b25e37342a0256cc75bd99d5392289082319f2378b6bed20c13c Jul 14 22:17:00.271638 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.271605 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:17:00.271734 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.271672 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:17:00.271734 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.271704 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:17:00.878621 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.878583 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" event={"ID":"04248810-5eb4-4aba-8ce3-006dc32f0b8c","Type":"ContainerStarted","Data":"2637382c9e068b3fa38b04eba3ddffeee6c96320e5656691d8a9c4dc0b671aaf"} Jul 14 22:17:00.879013 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.878643 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" event={"ID":"04248810-5eb4-4aba-8ce3-006dc32f0b8c","Type":"ContainerStarted","Data":"8175bb00a946b25e37342a0256cc75bd99d5392289082319f2378b6bed20c13c"} Jul 14 22:17:00.907364 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:00.907310 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" podStartSLOduration=1.9072895079999999 podStartE2EDuration="1.907289508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:17:00.903213144 +0000 UTC m=+1555.752683202" watchObservedRunningTime="2026-07-14 22:17:00.907289508 +0000 UTC m=+1555.756759548" Jul 14 22:17:01.123952 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:01.123907 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:01.129063 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:01.128971 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:01.884208 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:01.884178 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:01.885262 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:01.885242 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:06.007785 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.007742 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82"] Jul 14 22:17:06.011401 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.011374 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" Jul 14 22:17:06.014232 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.014205 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:17:06.014356 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.014239 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:17:06.026036 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.025997 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82"] Jul 14 22:17:06.054377 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.054341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50af7fd7-b910-4612-b039-58c741f7fc8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5574436c-8cc82\" (UID: \"50af7fd7-b910-4612-b039-58c741f7fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" Jul 14 22:17:06.155890 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.155777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50af7fd7-b910-4612-b039-58c741f7fc8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5574436c-8cc82\" (UID: \"50af7fd7-b910-4612-b039-58c741f7fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" Jul 14 22:17:06.160801 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.156393 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50af7fd7-b910-4612-b039-58c741f7fc8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5574436c-8cc82\" (UID: \"50af7fd7-b910-4612-b039-58c741f7fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" Jul 14 22:17:06.322089 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.322051 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" Jul 14 22:17:06.462058 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.462033 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82"] Jul 14 22:17:06.463205 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:17:06.463178 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50af7fd7_b910_4612_b039_58c741f7fc8a.slice/crio-11a531c4b6d77e7ff3c204e433701e8b3c52b3542955437aa9b7879ff81e7a36 WatchSource:0}: Error finding container 11a531c4b6d77e7ff3c204e433701e8b3c52b3542955437aa9b7879ff81e7a36: Status 404 returned error can't find the container with id 11a531c4b6d77e7ff3c204e433701e8b3c52b3542955437aa9b7879ff81e7a36 Jul 14 22:17:06.912979 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.912945 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" event={"ID":"50af7fd7-b910-4612-b039-58c741f7fc8a","Type":"ContainerStarted","Data":"3162d7dd28bb0c0cffb27252950cb9fd642b453ff4d5508468fa97f50a9b251b"} Jul 14 22:17:06.913200 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.912995 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" event={"ID":"50af7fd7-b910-4612-b039-58c741f7fc8a","Type":"ContainerStarted","Data":"11a531c4b6d77e7ff3c204e433701e8b3c52b3542955437aa9b7879ff81e7a36"} Jul 14 22:17:06.929592 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:06.929544 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" podStartSLOduration=1.9295287110000001 podStartE2EDuration="1.929528711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:17:06.928835084 +0000 UTC m=+1561.778305124" watchObservedRunningTime="2026-07-14 22:17:06.929528711 +0000 UTC m=+1561.778998749" Jul 14 22:17:07.919548 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:07.919514 2562 generic.go:356] "Generic (PLEG): container finished" podID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerID="3162d7dd28bb0c0cffb27252950cb9fd642b453ff4d5508468fa97f50a9b251b" exitCode=7 Jul 14 22:17:07.919936 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:07.919596 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" event={"ID":"50af7fd7-b910-4612-b039-58c741f7fc8a","Type":"ContainerDied","Data":"3162d7dd28bb0c0cffb27252950cb9fd642b453ff4d5508468fa97f50a9b251b"} Jul 14 22:17:07.919936 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:07.919810 2562 scope.go:117] "RemoveContainer" containerID="3162d7dd28bb0c0cffb27252950cb9fd642b453ff4d5508468fa97f50a9b251b" Jul 14 22:17:08.926703 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:08.926612 2562 generic.go:356] "Generic (PLEG): container finished" podID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerID="8b9651405e2123b37b5ffc799a49e19cb96bdaa3f380fd8690435c32699b7e7d" exitCode=7 Jul 14 22:17:08.927121 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:08.926697 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" event={"ID":"50af7fd7-b910-4612-b039-58c741f7fc8a","Type":"ContainerDied","Data":"8b9651405e2123b37b5ffc799a49e19cb96bdaa3f380fd8690435c32699b7e7d"} Jul 14 22:17:08.927121 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:08.926756 2562 scope.go:117] "RemoveContainer" containerID="3162d7dd28bb0c0cffb27252950cb9fd642b453ff4d5508468fa97f50a9b251b" Jul 14 22:17:08.927121 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:08.926966 2562 scope.go:117] "RemoveContainer" containerID="8b9651405e2123b37b5ffc799a49e19cb96bdaa3f380fd8690435c32699b7e7d" Jul 14 22:17:08.927260 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:17:08.927237 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5574436c-8cc82_odh-ai-gateway-infra(50af7fd7-b910-4612-b039-58c741f7fc8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" Jul 14 22:17:09.933591 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:09.933557 2562 scope.go:117] "RemoveContainer" containerID="8b9651405e2123b37b5ffc799a49e19cb96bdaa3f380fd8690435c32699b7e7d" Jul 14 22:17:09.934011 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:17:09.933788 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5574436c-8cc82_odh-ai-gateway-infra(50af7fd7-b910-4612-b039-58c741f7fc8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" Jul 14 22:17:16.886317 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:16.886279 2562 patch_prober.go:28] interesting pod/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 22:17:16.886742 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:16.886347 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 22:17:16.976174 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:16.976138 2562 generic.go:356] "Generic (PLEG): container finished" podID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerID="2637382c9e068b3fa38b04eba3ddffeee6c96320e5656691d8a9c4dc0b671aaf" exitCode=0 Jul 14 22:17:16.976370 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:16.976212 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" event={"ID":"04248810-5eb4-4aba-8ce3-006dc32f0b8c","Type":"ContainerDied","Data":"2637382c9e068b3fa38b04eba3ddffeee6c96320e5656691d8a9c4dc0b671aaf"} Jul 14 22:17:16.976637 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:16.976618 2562 scope.go:117] "RemoveContainer" containerID="2637382c9e068b3fa38b04eba3ddffeee6c96320e5656691d8a9c4dc0b671aaf" Jul 14 22:17:16.977632 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:16.977606 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:17:16.977710 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:16.977672 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:17:16.977710 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:16.977703 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:17:17.986115 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:17.986065 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" event={"ID":"04248810-5eb4-4aba-8ce3-006dc32f0b8c","Type":"ContainerStarted","Data":"7dd7d32f541112020eebb517932d1d623cf7e64d41a71ee8415134c269cfbd6d"} Jul 14 22:17:18.123369 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:18.123329 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:18.123583 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:18.123563 2562 patch_prober.go:28] interesting pod/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 22:17:18.123630 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:18.123608 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 22:17:19.123500 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:19.123463 2562 patch_prober.go:28] interesting pod/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 22:17:19.123918 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:19.123524 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 22:17:20.124003 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:20.123963 2562 patch_prober.go:28] interesting pod/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 22:17:20.124460 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:20.124058 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 22:17:21.124043 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:21.123977 2562 patch_prober.go:28] interesting pod/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 22:17:21.124490 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:21.124072 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 22:17:22.127334 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:22.127303 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:22.127787 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:22.127592 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:22.128426 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:22.128401 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:17:22.804990 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:22.804955 2562 scope.go:117] "RemoveContainer" containerID="8b9651405e2123b37b5ffc799a49e19cb96bdaa3f380fd8690435c32699b7e7d" Jul 14 22:17:23.023421 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:23.023394 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" event={"ID":"50af7fd7-b910-4612-b039-58c741f7fc8a","Type":"ContainerStarted","Data":"0f41c99f44119770f30a7e624696e4dc678f88f8d3e014136782004dd6d9dcbd"} Jul 14 22:17:24.030412 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:24.030380 2562 generic.go:356] "Generic (PLEG): container finished" podID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerID="0f41c99f44119770f30a7e624696e4dc678f88f8d3e014136782004dd6d9dcbd" exitCode=0 Jul 14 22:17:24.030842 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:24.030450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" event={"ID":"50af7fd7-b910-4612-b039-58c741f7fc8a","Type":"ContainerDied","Data":"0f41c99f44119770f30a7e624696e4dc678f88f8d3e014136782004dd6d9dcbd"} Jul 14 22:17:24.030842 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:24.030513 2562 scope.go:117] "RemoveContainer" containerID="8b9651405e2123b37b5ffc799a49e19cb96bdaa3f380fd8690435c32699b7e7d" Jul 14 22:17:24.055379 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:24.055355 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82"] Jul 14 22:17:25.179451 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:25.179425 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" Jul 14 22:17:25.342577 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:25.342488 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50af7fd7-b910-4612-b039-58c741f7fc8a-maas-api-service-ca\") pod \"50af7fd7-b910-4612-b039-58c741f7fc8a\" (UID: \"50af7fd7-b910-4612-b039-58c741f7fc8a\") " Jul 14 22:17:25.342850 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:25.342828 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50af7fd7-b910-4612-b039-58c741f7fc8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50af7fd7-b910-4612-b039-58c741f7fc8a" (UID: "50af7fd7-b910-4612-b039-58c741f7fc8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:17:25.443521 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:25.443481 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50af7fd7-b910-4612-b039-58c741f7fc8a-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:17:26.043822 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:26.043792 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" Jul 14 22:17:26.044016 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:26.043826 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82" event={"ID":"50af7fd7-b910-4612-b039-58c741f7fc8a","Type":"ContainerDied","Data":"11a531c4b6d77e7ff3c204e433701e8b3c52b3542955437aa9b7879ff81e7a36"} Jul 14 22:17:26.044016 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:26.043879 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11a531c4b6d77e7ff3c204e433701e8b3c52b3542955437aa9b7879ff81e7a36" Jul 14 22:17:26.076374 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:26.076344 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82"] Jul 14 22:17:26.078410 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:26.078391 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-8cc82"] Jul 14 22:17:26.810441 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:17:26.810406 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" path="/var/lib/kubelet/pods/50af7fd7-b910-4612-b039-58c741f7fc8a/volumes" Jul 14 22:18:06.048905 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.048813 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn"] Jul 14 22:18:06.049467 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.049215 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerName="revoke-keys" Jul 14 22:18:06.049467 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.049229 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerName="revoke-keys" Jul 14 22:18:06.049467 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.049262 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerName="revoke-keys" Jul 14 22:18:06.049467 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.049267 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerName="revoke-keys" Jul 14 22:18:06.049467 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.049329 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerName="revoke-keys" Jul 14 22:18:06.049467 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.049342 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerName="revoke-keys" Jul 14 22:18:06.052463 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.052446 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn" Jul 14 22:18:06.055196 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.055164 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:18:06.055318 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.055217 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:18:06.060477 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.060444 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn"] Jul 14 22:18:06.204109 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.204072 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d8ee8-7ee9-4ea1-a04f-d18a61110353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5574436c-g5gtn\" (UID: \"6a3d8ee8-7ee9-4ea1-a04f-d18a61110353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn" Jul 14 22:18:06.305053 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.304921 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d8ee8-7ee9-4ea1-a04f-d18a61110353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5574436c-g5gtn\" (UID: \"6a3d8ee8-7ee9-4ea1-a04f-d18a61110353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn" Jul 14 22:18:06.305593 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.305572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d8ee8-7ee9-4ea1-a04f-d18a61110353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5574436c-g5gtn\" (UID: \"6a3d8ee8-7ee9-4ea1-a04f-d18a61110353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn" Jul 14 22:18:06.364767 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.364732 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn" Jul 14 22:18:06.501273 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:06.501208 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn"] Jul 14 22:18:06.503719 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:18:06.503681 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3d8ee8_7ee9_4ea1_a04f_d18a61110353.slice/crio-f92fc87c7fa7c5c8bba05733afb45be915ca049bf7d4b1bd44aaa10ac6f24641 WatchSource:0}: Error finding container f92fc87c7fa7c5c8bba05733afb45be915ca049bf7d4b1bd44aaa10ac6f24641: Status 404 returned error can't find the container with id f92fc87c7fa7c5c8bba05733afb45be915ca049bf7d4b1bd44aaa10ac6f24641 Jul 14 22:18:07.280439 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:07.280398 2562 generic.go:356] "Generic (PLEG): container finished" podID="6a3d8ee8-7ee9-4ea1-a04f-d18a61110353" containerID="d77cda3403ec112de54ed1fcfea96659cf8a9c8e1b6cd6e9145a3a3d61737263" exitCode=0 Jul 14 22:18:07.280815 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:07.280450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn" event={"ID":"6a3d8ee8-7ee9-4ea1-a04f-d18a61110353","Type":"ContainerDied","Data":"d77cda3403ec112de54ed1fcfea96659cf8a9c8e1b6cd6e9145a3a3d61737263"} Jul 14 22:18:07.280815 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:07.280502 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn" event={"ID":"6a3d8ee8-7ee9-4ea1-a04f-d18a61110353","Type":"ContainerStarted","Data":"f92fc87c7fa7c5c8bba05733afb45be915ca049bf7d4b1bd44aaa10ac6f24641"} Jul 14 22:18:08.417828 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:08.417804 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn" Jul 14 22:18:08.524857 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:08.524827 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d8ee8-7ee9-4ea1-a04f-d18a61110353-maas-api-service-ca\") pod \"6a3d8ee8-7ee9-4ea1-a04f-d18a61110353\" (UID: \"6a3d8ee8-7ee9-4ea1-a04f-d18a61110353\") " Jul 14 22:18:08.525222 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:08.525200 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3d8ee8-7ee9-4ea1-a04f-d18a61110353-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a3d8ee8-7ee9-4ea1-a04f-d18a61110353" (UID: "6a3d8ee8-7ee9-4ea1-a04f-d18a61110353"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:18:08.625763 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:08.625674 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d8ee8-7ee9-4ea1-a04f-d18a61110353-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:09.291495 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:09.291456 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn" Jul 14 22:18:09.291674 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:09.291465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5574436c-g5gtn" event={"ID":"6a3d8ee8-7ee9-4ea1-a04f-d18a61110353","Type":"ContainerDied","Data":"f92fc87c7fa7c5c8bba05733afb45be915ca049bf7d4b1bd44aaa10ac6f24641"} Jul 14 22:18:09.291674 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:09.291598 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f92fc87c7fa7c5c8bba05733afb45be915ca049bf7d4b1bd44aaa10ac6f24641" Jul 14 22:18:31.297990 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.297950 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf"] Jul 14 22:18:31.298847 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.298263 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" containerID="cri-o://7dd7d32f541112020eebb517932d1d623cf7e64d41a71ee8415134c269cfbd6d" gracePeriod=30 Jul 14 22:18:31.864689 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.864651 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz"] Jul 14 22:18:31.865070 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.865054 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a3d8ee8-7ee9-4ea1-a04f-d18a61110353" containerName="revoke-keys" Jul 14 22:18:31.865175 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.865090 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a3d8ee8-7ee9-4ea1-a04f-d18a61110353" containerName="revoke-keys" Jul 14 22:18:31.865175 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.865114 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerName="revoke-keys" Jul 14 22:18:31.865175 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.865119 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerName="revoke-keys" Jul 14 22:18:31.865334 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.865187 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="50af7fd7-b910-4612-b039-58c741f7fc8a" containerName="revoke-keys" Jul 14 22:18:31.865334 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.865198 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a3d8ee8-7ee9-4ea1-a04f-d18a61110353" containerName="revoke-keys" Jul 14 22:18:31.868471 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.868450 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:31.871017 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.870988 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-de92a977-gw-openshift-default-dockercfg-bjs5b\"" Jul 14 22:18:31.882476 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.882452 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz"] Jul 14 22:18:31.932133 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.932101 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhln7\" (UniqueName: \"kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-kube-api-access-zhln7\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:31.932133 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.932143 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99360d4-9def-44d7-bd18-91622860a7a7-istiod-ca-cert\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:31.932356 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.932165 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-credential-socket\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:31.932356 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.932208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-istio-token\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:31.932356 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.932228 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-certs\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:31.932356 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.932244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-envoy\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:31.932356 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.932281 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99360d4-9def-44d7-bd18-91622860a7a7-istio-podinfo\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:31.932356 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.932299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-socket\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:31.932356 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:31.932313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-data\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.032840 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.032807 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99360d4-9def-44d7-bd18-91622860a7a7-istio-podinfo\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.032840 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.032845 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-socket\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033094 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.032868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-data\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033094 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.032933 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhln7\" (UniqueName: \"kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-kube-api-access-zhln7\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033094 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.032975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99360d4-9def-44d7-bd18-91622860a7a7-istiod-ca-cert\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033094 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.033008 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-credential-socket\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033094 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.033051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-istio-token\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033094 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.033083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-certs\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033390 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.033113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-envoy\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033390 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.033315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-data\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033652 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.033615 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-credential-socket\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033755 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.033650 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-certs\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033755 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.033725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99360d4-9def-44d7-bd18-91622860a7a7-istiod-ca-cert\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.033864 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.033752 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-socket\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.035410 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.035386 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-envoy\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.035535 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.035518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99360d4-9def-44d7-bd18-91622860a7a7-istio-podinfo\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.044727 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.044707 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-istio-token\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.045641 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.045622 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhln7\" (UniqueName: \"kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-kube-api-access-zhln7\") pod \"e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.179742 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.179643 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:32.328151 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.328121 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz"] Jul 14 22:18:32.329172 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:18:32.329147 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99360d4_9def_44d7_bd18_91622860a7a7.slice/crio-5f438121a9d50852a343e7d5d25f0d3fdec3cb5717307c625d895ee64e9b19bd WatchSource:0}: Error finding container 5f438121a9d50852a343e7d5d25f0d3fdec3cb5717307c625d895ee64e9b19bd: Status 404 returned error can't find the container with id 5f438121a9d50852a343e7d5d25f0d3fdec3cb5717307c625d895ee64e9b19bd Jul 14 22:18:32.331332 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.331296 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:18:32.331419 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.331371 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:18:32.331419 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.331401 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:18:32.423008 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.422967 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" event={"ID":"c99360d4-9def-44d7-bd18-91622860a7a7","Type":"ContainerStarted","Data":"bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8"} Jul 14 22:18:32.423208 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.423050 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" event={"ID":"c99360d4-9def-44d7-bd18-91622860a7a7","Type":"ContainerStarted","Data":"5f438121a9d50852a343e7d5d25f0d3fdec3cb5717307c625d895ee64e9b19bd"} Jul 14 22:18:32.446268 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:32.446157 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" podStartSLOduration=1.44614095 podStartE2EDuration="1.44614095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:18:32.443395949 +0000 UTC m=+1647.292865981" watchObservedRunningTime="2026-07-14 22:18:32.44614095 +0000 UTC m=+1647.295610989" Jul 14 22:18:33.180774 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:33.180738 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:33.185541 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:33.185517 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:33.428541 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:33.428511 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:33.429615 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:33.429593 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:36.450049 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.449992 2562 generic.go:356] "Generic (PLEG): container finished" podID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerID="7dd7d32f541112020eebb517932d1d623cf7e64d41a71ee8415134c269cfbd6d" exitCode=0 Jul 14 22:18:36.450510 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.450069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" event={"ID":"04248810-5eb4-4aba-8ce3-006dc32f0b8c","Type":"ContainerDied","Data":"7dd7d32f541112020eebb517932d1d623cf7e64d41a71ee8415134c269cfbd6d"} Jul 14 22:18:36.450510 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.450143 2562 scope.go:117] "RemoveContainer" containerID="2637382c9e068b3fa38b04eba3ddffeee6c96320e5656691d8a9c4dc0b671aaf" Jul 14 22:18:36.559203 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.559144 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:18:36.576681 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.576650 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istiod-ca-cert\") pod \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " Jul 14 22:18:36.576845 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.576710 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-certs\") pod \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " Jul 14 22:18:36.576845 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.576734 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-podinfo\") pod \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " Jul 14 22:18:36.576845 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.576749 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-token\") pod \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " Jul 14 22:18:36.576845 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.576772 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-envoy\") pod \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " Jul 14 22:18:36.576845 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.576804 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-credential-socket\") pod \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " Jul 14 22:18:36.576845 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.576824 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-data\") pod \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " Jul 14 22:18:36.577182 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.576879 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-socket\") pod \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " Jul 14 22:18:36.577182 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.576957 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzthm\" (UniqueName: \"kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-kube-api-access-mzthm\") pod \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\" (UID: \"04248810-5eb4-4aba-8ce3-006dc32f0b8c\") " Jul 14 22:18:36.577182 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.577071 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04248810-5eb4-4aba-8ce3-006dc32f0b8c" (UID: "04248810-5eb4-4aba-8ce3-006dc32f0b8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:36.577182 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.577086 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04248810-5eb4-4aba-8ce3-006dc32f0b8c" (UID: "04248810-5eb4-4aba-8ce3-006dc32f0b8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:18:36.577395 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.577327 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:36.577395 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.577349 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:36.577395 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.577329 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04248810-5eb4-4aba-8ce3-006dc32f0b8c" (UID: "04248810-5eb4-4aba-8ce3-006dc32f0b8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:36.577670 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.577629 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04248810-5eb4-4aba-8ce3-006dc32f0b8c" (UID: "04248810-5eb4-4aba-8ce3-006dc32f0b8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:36.577922 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.577904 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-data" (OuterVolumeSpecName: "istio-data") pod "04248810-5eb4-4aba-8ce3-006dc32f0b8c" (UID: "04248810-5eb4-4aba-8ce3-006dc32f0b8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:36.579764 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.579738 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04248810-5eb4-4aba-8ce3-006dc32f0b8c" (UID: "04248810-5eb4-4aba-8ce3-006dc32f0b8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:18:36.579764 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.579755 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-token" (OuterVolumeSpecName: "istio-token") pod "04248810-5eb4-4aba-8ce3-006dc32f0b8c" (UID: "04248810-5eb4-4aba-8ce3-006dc32f0b8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:18:36.579924 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.579790 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04248810-5eb4-4aba-8ce3-006dc32f0b8c" (UID: "04248810-5eb4-4aba-8ce3-006dc32f0b8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:36.579924 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.579848 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-kube-api-access-mzthm" (OuterVolumeSpecName: "kube-api-access-mzthm") pod "04248810-5eb4-4aba-8ce3-006dc32f0b8c" (UID: "04248810-5eb4-4aba-8ce3-006dc32f0b8c"). InnerVolumeSpecName "kube-api-access-mzthm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:18:36.678004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.677965 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:36.678004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.677998 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:36.678004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.678007 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:36.678004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.678017 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:36.678307 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.678039 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:36.678307 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.678047 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04248810-5eb4-4aba-8ce3-006dc32f0b8c-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:36.678307 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:36.678057 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzthm\" (UniqueName: \"kubernetes.io/projected/04248810-5eb4-4aba-8ce3-006dc32f0b8c-kube-api-access-mzthm\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:37.456637 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.456609 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" Jul 14 22:18:37.457071 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.456607 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf" event={"ID":"04248810-5eb4-4aba-8ce3-006dc32f0b8c","Type":"ContainerDied","Data":"8175bb00a946b25e37342a0256cc75bd99d5392289082319f2378b6bed20c13c"} Jul 14 22:18:37.457071 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.456744 2562 scope.go:117] "RemoveContainer" containerID="7dd7d32f541112020eebb517932d1d623cf7e64d41a71ee8415134c269cfbd6d" Jul 14 22:18:37.476123 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.476093 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf"] Jul 14 22:18:37.479667 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.479640 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5574436c-openshift-default-6d6695646f-whqrf"] Jul 14 22:18:37.684088 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.682491 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97"] Jul 14 22:18:37.705043 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.701157 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" Jul 14 22:18:37.705043 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.701196 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" Jul 14 22:18:37.705043 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.701387 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" Jul 14 22:18:37.705043 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.701400 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" Jul 14 22:18:37.712089 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.706292 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" Jul 14 22:18:37.712089 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.711788 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97"] Jul 14 22:18:37.716043 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.715546 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:18:37.716043 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.715892 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:18:37.795049 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.791849 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a7d458-7178-41b0-a169-fa5ef141af97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-de92a977-gbf97\" (UID: \"22a7d458-7178-41b0-a169-fa5ef141af97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" Jul 14 22:18:37.892655 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.892600 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a7d458-7178-41b0-a169-fa5ef141af97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-de92a977-gbf97\" (UID: \"22a7d458-7178-41b0-a169-fa5ef141af97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" Jul 14 22:18:37.893650 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:37.893621 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a7d458-7178-41b0-a169-fa5ef141af97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-de92a977-gbf97\" (UID: \"22a7d458-7178-41b0-a169-fa5ef141af97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" Jul 14 22:18:38.025255 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:38.025213 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" Jul 14 22:18:38.210252 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:38.209816 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97"] Jul 14 22:18:38.210551 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:18:38.210509 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a7d458_7178_41b0_a169_fa5ef141af97.slice/crio-2530f64f14e53b6f5eff58801fc51d3de86cd6523448dadba12b899fe8bf341a WatchSource:0}: Error finding container 2530f64f14e53b6f5eff58801fc51d3de86cd6523448dadba12b899fe8bf341a: Status 404 returned error can't find the container with id 2530f64f14e53b6f5eff58801fc51d3de86cd6523448dadba12b899fe8bf341a Jul 14 22:18:38.463857 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:38.463821 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" event={"ID":"22a7d458-7178-41b0-a169-fa5ef141af97","Type":"ContainerStarted","Data":"2530f64f14e53b6f5eff58801fc51d3de86cd6523448dadba12b899fe8bf341a"} Jul 14 22:18:38.813069 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:38.813016 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" path="/var/lib/kubelet/pods/04248810-5eb4-4aba-8ce3-006dc32f0b8c/volumes" Jul 14 22:18:39.471429 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:39.471341 2562 generic.go:356] "Generic (PLEG): container finished" podID="22a7d458-7178-41b0-a169-fa5ef141af97" containerID="ba1833501b10ae3aae26682eae450f81570e15f3521494148df8a10af76db287" exitCode=7 Jul 14 22:18:39.471808 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:39.471427 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" event={"ID":"22a7d458-7178-41b0-a169-fa5ef141af97","Type":"ContainerDied","Data":"ba1833501b10ae3aae26682eae450f81570e15f3521494148df8a10af76db287"} Jul 14 22:18:39.471808 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:39.471585 2562 scope.go:117] "RemoveContainer" containerID="ba1833501b10ae3aae26682eae450f81570e15f3521494148df8a10af76db287" Jul 14 22:18:40.478455 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:40.478417 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" event={"ID":"22a7d458-7178-41b0-a169-fa5ef141af97","Type":"ContainerStarted","Data":"1090ceabab44f490e028d5ea79e78bc0af130eaa48334f5374e6c0916b701c00"} Jul 14 22:18:40.498165 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:40.498100 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" podStartSLOduration=3.498073029 podStartE2EDuration="3.498073029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:18:40.495558641 +0000 UTC m=+1655.345028682" watchObservedRunningTime="2026-07-14 22:18:40.498073029 +0000 UTC m=+1655.347543068" Jul 14 22:18:41.485653 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:41.485555 2562 generic.go:356] "Generic (PLEG): container finished" podID="22a7d458-7178-41b0-a169-fa5ef141af97" containerID="1090ceabab44f490e028d5ea79e78bc0af130eaa48334f5374e6c0916b701c00" exitCode=7 Jul 14 22:18:41.486061 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:41.485650 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" event={"ID":"22a7d458-7178-41b0-a169-fa5ef141af97","Type":"ContainerDied","Data":"1090ceabab44f490e028d5ea79e78bc0af130eaa48334f5374e6c0916b701c00"} Jul 14 22:18:41.486061 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:41.485722 2562 scope.go:117] "RemoveContainer" containerID="ba1833501b10ae3aae26682eae450f81570e15f3521494148df8a10af76db287" Jul 14 22:18:41.486061 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:41.485778 2562 scope.go:117] "RemoveContainer" containerID="1090ceabab44f490e028d5ea79e78bc0af130eaa48334f5374e6c0916b701c00" Jul 14 22:18:41.486061 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:18:41.486051 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-de92a977-gbf97_odh-ai-gateway-infra(22a7d458-7178-41b0-a169-fa5ef141af97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" Jul 14 22:18:42.493785 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:42.493752 2562 scope.go:117] "RemoveContainer" containerID="1090ceabab44f490e028d5ea79e78bc0af130eaa48334f5374e6c0916b701c00" Jul 14 22:18:42.494336 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:18:42.493956 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-de92a977-gbf97_odh-ai-gateway-infra(22a7d458-7178-41b0-a169-fa5ef141af97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" Jul 14 22:18:44.505399 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:44.505363 2562 generic.go:356] "Generic (PLEG): container finished" podID="c99360d4-9def-44d7-bd18-91622860a7a7" containerID="bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8" exitCode=0 Jul 14 22:18:44.505881 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:44.505438 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" event={"ID":"c99360d4-9def-44d7-bd18-91622860a7a7","Type":"ContainerDied","Data":"bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8"} Jul 14 22:18:44.505881 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:44.505835 2562 scope.go:117] "RemoveContainer" containerID="bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8" Jul 14 22:18:44.506832 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:44.506804 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:18:44.506908 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:44.506890 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:18:44.506947 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:44.506934 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:18:45.513272 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:45.513236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" event={"ID":"c99360d4-9def-44d7-bd18-91622860a7a7","Type":"ContainerStarted","Data":"8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e"} Jul 14 22:18:46.180757 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:46.180717 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:46.181847 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:46.181819 2562 patch_prober.go:28] interesting pod/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 22:18:46.181968 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:46.181878 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 22:18:47.180294 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:47.180255 2562 patch_prober.go:28] interesting pod/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 22:18:47.180733 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:47.180314 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 22:18:48.180388 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:48.180350 2562 patch_prober.go:28] interesting pod/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 22:18:48.180793 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:48.180418 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 22:18:48.429389 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:48.429347 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:49.180771 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:49.180724 2562 patch_prober.go:28] interesting pod/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 22:18:49.181295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:49.180800 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 22:18:50.184377 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:50.184343 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:50.185451 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:50.185430 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:18:55.804843 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:55.804806 2562 scope.go:117] "RemoveContainer" containerID="1090ceabab44f490e028d5ea79e78bc0af130eaa48334f5374e6c0916b701c00" Jul 14 22:18:56.573088 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:56.573053 2562 generic.go:356] "Generic (PLEG): container finished" podID="22a7d458-7178-41b0-a169-fa5ef141af97" containerID="5816a226e390c7cd0ab537d1b490bc1797beea2fc306051eacd6328261749ca0" exitCode=0 Jul 14 22:18:56.573254 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:56.573051 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" event={"ID":"22a7d458-7178-41b0-a169-fa5ef141af97","Type":"ContainerDied","Data":"5816a226e390c7cd0ab537d1b490bc1797beea2fc306051eacd6328261749ca0"} Jul 14 22:18:56.573254 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:56.573206 2562 scope.go:117] "RemoveContainer" containerID="1090ceabab44f490e028d5ea79e78bc0af130eaa48334f5374e6c0916b701c00" Jul 14 22:18:57.605944 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:57.605907 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97"] Jul 14 22:18:57.719986 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:57.719956 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" Jul 14 22:18:57.794778 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:57.794737 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a7d458-7178-41b0-a169-fa5ef141af97-maas-api-service-ca\") pod \"22a7d458-7178-41b0-a169-fa5ef141af97\" (UID: \"22a7d458-7178-41b0-a169-fa5ef141af97\") " Jul 14 22:18:57.795196 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:57.795175 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a7d458-7178-41b0-a169-fa5ef141af97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22a7d458-7178-41b0-a169-fa5ef141af97" (UID: "22a7d458-7178-41b0-a169-fa5ef141af97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:18:57.896046 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:57.895926 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a7d458-7178-41b0-a169-fa5ef141af97-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:18:58.587847 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:58.587815 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" Jul 14 22:18:58.588052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:58.587858 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97" event={"ID":"22a7d458-7178-41b0-a169-fa5ef141af97","Type":"ContainerDied","Data":"2530f64f14e53b6f5eff58801fc51d3de86cd6523448dadba12b899fe8bf341a"} Jul 14 22:18:58.588052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:58.587907 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2530f64f14e53b6f5eff58801fc51d3de86cd6523448dadba12b899fe8bf341a" Jul 14 22:18:58.623416 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:58.623384 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97"] Jul 14 22:18:58.625520 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:58.625498 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-gbf97"] Jul 14 22:18:58.810541 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:18:58.810506 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" path="/var/lib/kubelet/pods/22a7d458-7178-41b0-a169-fa5ef141af97/volumes" Jul 14 22:19:37.730011 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.729915 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb"] Jul 14 22:19:37.730612 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.730452 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" containerName="revoke-keys" Jul 14 22:19:37.730612 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.730471 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" containerName="revoke-keys" Jul 14 22:19:37.730612 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.730494 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" containerName="revoke-keys" Jul 14 22:19:37.730612 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.730503 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" containerName="revoke-keys" Jul 14 22:19:37.730612 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.730514 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" containerName="revoke-keys" Jul 14 22:19:37.730612 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.730523 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" containerName="revoke-keys" Jul 14 22:19:37.730612 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.730552 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" Jul 14 22:19:37.730612 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.730560 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04248810-5eb4-4aba-8ce3-006dc32f0b8c" containerName="istio-proxy" Jul 14 22:19:37.731016 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.730650 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" containerName="revoke-keys" Jul 14 22:19:37.731016 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.730668 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" containerName="revoke-keys" Jul 14 22:19:37.733670 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.733649 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" Jul 14 22:19:37.736271 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.736252 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:19:37.736381 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.736257 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:19:37.742467 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.742435 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb"] Jul 14 22:19:37.825118 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.825084 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz"] Jul 14 22:19:37.825323 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.825283 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" containerID="cri-o://8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e" gracePeriod=30 Jul 14 22:19:37.862189 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.862150 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c973853-901f-47cd-a000-1159937992e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb\" (UID: \"6c973853-901f-47cd-a000-1159937992e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" Jul 14 22:19:37.963414 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.963381 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c973853-901f-47cd-a000-1159937992e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb\" (UID: \"6c973853-901f-47cd-a000-1159937992e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" Jul 14 22:19:37.964255 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:37.964222 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c973853-901f-47cd-a000-1159937992e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb\" (UID: \"6c973853-901f-47cd-a000-1159937992e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" Jul 14 22:19:38.044964 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:38.044933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" Jul 14 22:19:38.385607 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:38.385584 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb"] Jul 14 22:19:38.388518 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:19:38.388493 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c973853_901f_47cd_a000_1159937992e3.slice/crio-19ab1ef12aad6ec721d46f599c77cd2e3e7b11e91c33836b88da69f946fc57bd WatchSource:0}: Error finding container 19ab1ef12aad6ec721d46f599c77cd2e3e7b11e91c33836b88da69f946fc57bd: Status 404 returned error can't find the container with id 19ab1ef12aad6ec721d46f599c77cd2e3e7b11e91c33836b88da69f946fc57bd Jul 14 22:19:38.801694 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:38.801661 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" event={"ID":"6c973853-901f-47cd-a000-1159937992e3","Type":"ContainerStarted","Data":"408d5ed786408e963433a8378befdabb725fd2d07a2773f137221307d99806d5"} Jul 14 22:19:38.802098 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:38.801713 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" event={"ID":"6c973853-901f-47cd-a000-1159937992e3","Type":"ContainerStarted","Data":"19ab1ef12aad6ec721d46f599c77cd2e3e7b11e91c33836b88da69f946fc57bd"} Jul 14 22:19:38.820426 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:38.820366 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" podStartSLOduration=1.820349916 podStartE2EDuration="1.820349916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:19:38.818253841 +0000 UTC m=+1713.667723881" watchObservedRunningTime="2026-07-14 22:19:38.820349916 +0000 UTC m=+1713.669819955" Jul 14 22:19:39.808175 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:39.808142 2562 generic.go:356] "Generic (PLEG): container finished" podID="6c973853-901f-47cd-a000-1159937992e3" containerID="408d5ed786408e963433a8378befdabb725fd2d07a2773f137221307d99806d5" exitCode=0 Jul 14 22:19:39.808595 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:39.808231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" event={"ID":"6c973853-901f-47cd-a000-1159937992e3","Type":"ContainerDied","Data":"408d5ed786408e963433a8378befdabb725fd2d07a2773f137221307d99806d5"} Jul 14 22:19:40.945559 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:40.945533 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" Jul 14 22:19:41.092602 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:41.092507 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c973853-901f-47cd-a000-1159937992e3-maas-api-service-ca\") pod \"6c973853-901f-47cd-a000-1159937992e3\" (UID: \"6c973853-901f-47cd-a000-1159937992e3\") " Jul 14 22:19:41.092871 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:41.092848 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c973853-901f-47cd-a000-1159937992e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c973853-901f-47cd-a000-1159937992e3" (UID: "6c973853-901f-47cd-a000-1159937992e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:19:41.194246 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:41.194209 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c973853-901f-47cd-a000-1159937992e3-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:19:41.820350 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:41.820320 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" Jul 14 22:19:41.820350 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:41.820330 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb" event={"ID":"6c973853-901f-47cd-a000-1159937992e3","Type":"ContainerDied","Data":"19ab1ef12aad6ec721d46f599c77cd2e3e7b11e91c33836b88da69f946fc57bd"} Jul 14 22:19:41.820576 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:41.820384 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19ab1ef12aad6ec721d46f599c77cd2e3e7b11e91c33836b88da69f946fc57bd" Jul 14 22:19:43.077753 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.077720 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:19:43.214137 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214104 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-credential-socket\") pod \"c99360d4-9def-44d7-bd18-91622860a7a7\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " Jul 14 22:19:43.214137 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214135 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-istio-token\") pod \"c99360d4-9def-44d7-bd18-91622860a7a7\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " Jul 14 22:19:43.214397 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214160 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99360d4-9def-44d7-bd18-91622860a7a7-istiod-ca-cert\") pod \"c99360d4-9def-44d7-bd18-91622860a7a7\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " Jul 14 22:19:43.214397 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214181 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-socket\") pod \"c99360d4-9def-44d7-bd18-91622860a7a7\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " Jul 14 22:19:43.214397 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214217 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-certs\") pod \"c99360d4-9def-44d7-bd18-91622860a7a7\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " Jul 14 22:19:43.214397 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214248 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-envoy\") pod \"c99360d4-9def-44d7-bd18-91622860a7a7\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " Jul 14 22:19:43.214397 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214298 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99360d4-9def-44d7-bd18-91622860a7a7-istio-podinfo\") pod \"c99360d4-9def-44d7-bd18-91622860a7a7\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " Jul 14 22:19:43.214397 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214320 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-data\") pod \"c99360d4-9def-44d7-bd18-91622860a7a7\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " Jul 14 22:19:43.214397 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214362 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhln7\" (UniqueName: \"kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-kube-api-access-zhln7\") pod \"c99360d4-9def-44d7-bd18-91622860a7a7\" (UID: \"c99360d4-9def-44d7-bd18-91622860a7a7\") " Jul 14 22:19:43.214397 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214383 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c99360d4-9def-44d7-bd18-91622860a7a7" (UID: "c99360d4-9def-44d7-bd18-91622860a7a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:19:43.214807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214505 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c99360d4-9def-44d7-bd18-91622860a7a7" (UID: "c99360d4-9def-44d7-bd18-91622860a7a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:19:43.214807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214556 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c99360d4-9def-44d7-bd18-91622860a7a7" (UID: "c99360d4-9def-44d7-bd18-91622860a7a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:19:43.214807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214592 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99360d4-9def-44d7-bd18-91622860a7a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c99360d4-9def-44d7-bd18-91622860a7a7" (UID: "c99360d4-9def-44d7-bd18-91622860a7a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:19:43.214807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214674 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:19:43.214807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214718 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99360d4-9def-44d7-bd18-91622860a7a7-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:19:43.214807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214741 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:19:43.214807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.214756 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:19:43.215526 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.215498 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "c99360d4-9def-44d7-bd18-91622860a7a7" (UID: "c99360d4-9def-44d7-bd18-91622860a7a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:19:43.216436 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.216412 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c99360d4-9def-44d7-bd18-91622860a7a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c99360d4-9def-44d7-bd18-91622860a7a7" (UID: "c99360d4-9def-44d7-bd18-91622860a7a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:19:43.216534 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.216519 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "c99360d4-9def-44d7-bd18-91622860a7a7" (UID: "c99360d4-9def-44d7-bd18-91622860a7a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:19:43.216606 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.216586 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c99360d4-9def-44d7-bd18-91622860a7a7" (UID: "c99360d4-9def-44d7-bd18-91622860a7a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:19:43.216644 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.216635 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-kube-api-access-zhln7" (OuterVolumeSpecName: "kube-api-access-zhln7") pod "c99360d4-9def-44d7-bd18-91622860a7a7" (UID: "c99360d4-9def-44d7-bd18-91622860a7a7"). InnerVolumeSpecName "kube-api-access-zhln7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:19:43.315228 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.315192 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhln7\" (UniqueName: \"kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-kube-api-access-zhln7\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:19:43.315228 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.315221 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99360d4-9def-44d7-bd18-91622860a7a7-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:19:43.315228 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.315231 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:19:43.315228 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.315239 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99360d4-9def-44d7-bd18-91622860a7a7-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:19:43.315510 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.315248 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99360d4-9def-44d7-bd18-91622860a7a7-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:19:43.832129 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.832096 2562 generic.go:356] "Generic (PLEG): container finished" podID="c99360d4-9def-44d7-bd18-91622860a7a7" containerID="8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e" exitCode=0 Jul 14 22:19:43.832312 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.832148 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" Jul 14 22:19:43.832312 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.832190 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" event={"ID":"c99360d4-9def-44d7-bd18-91622860a7a7","Type":"ContainerDied","Data":"8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e"} Jul 14 22:19:43.832312 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.832265 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz" event={"ID":"c99360d4-9def-44d7-bd18-91622860a7a7","Type":"ContainerDied","Data":"5f438121a9d50852a343e7d5d25f0d3fdec3cb5717307c625d895ee64e9b19bd"} Jul 14 22:19:43.832312 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.832294 2562 scope.go:117] "RemoveContainer" containerID="8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e" Jul 14 22:19:43.843604 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.843587 2562 scope.go:117] "RemoveContainer" containerID="bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8" Jul 14 22:19:43.851872 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.851854 2562 scope.go:117] "RemoveContainer" containerID="8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e" Jul 14 22:19:43.852162 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:19:43.852139 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e\": container with ID starting with 8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e not found: ID does not exist" containerID="8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e" Jul 14 22:19:43.852221 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.852171 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e"} err="failed to get container status \"8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e\": rpc error: code = NotFound desc = could not find container \"8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e\": container with ID starting with 8f08e680d765f0e34fc9f925c8e516f1a1baf055b8fd6f6fb9c6f97d8f27026e not found: ID does not exist" Jul 14 22:19:43.852221 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.852188 2562 scope.go:117] "RemoveContainer" containerID="bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8" Jul 14 22:19:43.852440 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:19:43.852419 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8\": container with ID starting with bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8 not found: ID does not exist" containerID="bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8" Jul 14 22:19:43.852495 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.852448 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8"} err="failed to get container status \"bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8\": rpc error: code = NotFound desc = could not find container \"bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8\": container with ID starting with bac557ce6437e828497c3f4c90fc26ea24c1189873c63b708220a6741e6c21b8 not found: ID does not exist" Jul 14 22:19:43.857760 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.857736 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz"] Jul 14 22:19:43.861467 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:43.861446 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-de92a977-gw-openshift-default-6b84676c47-298rz"] Jul 14 22:19:44.816139 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:44.816105 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" path="/var/lib/kubelet/pods/c99360d4-9def-44d7-bd18-91622860a7a7/volumes" Jul 14 22:19:50.034098 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.033999 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc"] Jul 14 22:19:50.034621 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.034594 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c973853-901f-47cd-a000-1159937992e3" containerName="revoke-keys" Jul 14 22:19:50.034621 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.034619 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c973853-901f-47cd-a000-1159937992e3" containerName="revoke-keys" Jul 14 22:19:50.034789 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.034640 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" Jul 14 22:19:50.034789 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.034647 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" Jul 14 22:19:50.034789 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.034724 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a7d458-7178-41b0-a169-fa5ef141af97" containerName="revoke-keys" Jul 14 22:19:50.034789 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.034734 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" Jul 14 22:19:50.034789 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.034745 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c973853-901f-47cd-a000-1159937992e3" containerName="revoke-keys" Jul 14 22:19:50.039461 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.039442 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.042555 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.042533 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c29be32a-openshift-default-dockercfg-pgdtx\"" Jul 14 22:19:50.052372 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.052344 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc"] Jul 14 22:19:50.177324 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.177274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-envoy\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.177324 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.177329 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf415728-e45e-4556-8290-eccbd3fba786-istio-podinfo\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.177543 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.177353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxl4\" (UniqueName: \"kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-kube-api-access-cvxl4\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.177543 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.177400 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-certs\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.177543 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.177418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-data\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.177543 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.177434 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-istio-token\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.177543 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.177525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-socket\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.177704 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.177570 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf415728-e45e-4556-8290-eccbd3fba786-istiod-ca-cert\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.177704 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.177611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-credential-socket\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.278477 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.278442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-certs\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.278477 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.278479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-data\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.278681 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.278498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-istio-token\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.278681 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.278537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-socket\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.278681 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.278566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf415728-e45e-4556-8290-eccbd3fba786-istiod-ca-cert\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.278681 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.278587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-credential-socket\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.278898 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.278683 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-envoy\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.278898 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.278737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf415728-e45e-4556-8290-eccbd3fba786-istio-podinfo\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.278898 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.278768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxl4\" (UniqueName: \"kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-kube-api-access-cvxl4\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.279091 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.278904 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-data\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.279164 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.279100 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-credential-socket\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.279164 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.279136 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-socket\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.279332 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.279313 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-certs\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.279416 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.279392 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf415728-e45e-4556-8290-eccbd3fba786-istiod-ca-cert\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.281091 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.281068 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-envoy\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.281315 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.281297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf415728-e45e-4556-8290-eccbd3fba786-istio-podinfo\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.286692 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.286629 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxl4\" (UniqueName: \"kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-kube-api-access-cvxl4\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.286969 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.286949 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-istio-token\") pod \"e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.351173 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.351138 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:50.499969 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.499945 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc"] Jul 14 22:19:50.502637 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:19:50.502608 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf415728_e45e_4556_8290_eccbd3fba786.slice/crio-bd3c0627eb76342163970ec65cf4e965eec04c7e882669299d62ff63f656c9ee WatchSource:0}: Error finding container bd3c0627eb76342163970ec65cf4e965eec04c7e882669299d62ff63f656c9ee: Status 404 returned error can't find the container with id bd3c0627eb76342163970ec65cf4e965eec04c7e882669299d62ff63f656c9ee Jul 14 22:19:50.504868 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.504832 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:19:50.504971 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.504918 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:19:50.505067 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.504969 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:19:50.873497 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.873405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" event={"ID":"cf415728-e45e-4556-8290-eccbd3fba786","Type":"ContainerStarted","Data":"362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0"} Jul 14 22:19:50.873497 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.873457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" event={"ID":"cf415728-e45e-4556-8290-eccbd3fba786","Type":"ContainerStarted","Data":"bd3c0627eb76342163970ec65cf4e965eec04c7e882669299d62ff63f656c9ee"} Jul 14 22:19:50.893891 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:50.893844 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" podStartSLOduration=0.893830293 podStartE2EDuration="893.830293ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:19:50.892814526 +0000 UTC m=+1725.742284578" watchObservedRunningTime="2026-07-14 22:19:50.893830293 +0000 UTC m=+1725.743300331" Jul 14 22:19:51.351798 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:51.351761 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:51.370865 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:51.370831 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:51.878577 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:51.878546 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:51.879639 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:51.879621 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:19:56.268105 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.268065 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf"] Jul 14 22:19:56.268488 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.268459 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" Jul 14 22:19:56.268488 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.268472 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" Jul 14 22:19:56.268573 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.268545 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c99360d4-9def-44d7-bd18-91622860a7a7" containerName="istio-proxy" Jul 14 22:19:56.271907 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.271891 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" Jul 14 22:19:56.274630 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.274601 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:19:56.275667 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.275640 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:19:56.279001 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.278975 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf"] Jul 14 22:19:56.332639 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.332602 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8d1246-4c67-4614-9ff6-e122b5d3c55e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf\" (UID: \"fa8d1246-4c67-4614-9ff6-e122b5d3c55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" Jul 14 22:19:56.433147 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.433098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8d1246-4c67-4614-9ff6-e122b5d3c55e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf\" (UID: \"fa8d1246-4c67-4614-9ff6-e122b5d3c55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" Jul 14 22:19:56.433793 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.433771 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8d1246-4c67-4614-9ff6-e122b5d3c55e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf\" (UID: \"fa8d1246-4c67-4614-9ff6-e122b5d3c55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" Jul 14 22:19:56.590403 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.590363 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" Jul 14 22:19:56.757615 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.757584 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf"] Jul 14 22:19:56.759331 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:19:56.759296 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8d1246_4c67_4614_9ff6_e122b5d3c55e.slice/crio-f89730df82eeae025044bcb217cd98df6e2ffac13a73d382b76ca719f9d57799 WatchSource:0}: Error finding container f89730df82eeae025044bcb217cd98df6e2ffac13a73d382b76ca719f9d57799: Status 404 returned error can't find the container with id f89730df82eeae025044bcb217cd98df6e2ffac13a73d382b76ca719f9d57799 Jul 14 22:19:56.912746 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:56.912659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" event={"ID":"fa8d1246-4c67-4614-9ff6-e122b5d3c55e","Type":"ContainerStarted","Data":"f89730df82eeae025044bcb217cd98df6e2ffac13a73d382b76ca719f9d57799"} Jul 14 22:19:57.920164 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:57.920122 2562 generic.go:356] "Generic (PLEG): container finished" podID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerID="a1f87ba8738cc681d1f3a9cc3c95a3a66859c9672b4f9af7322cbdbbb1fd651a" exitCode=7 Jul 14 22:19:57.920561 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:57.920214 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" event={"ID":"fa8d1246-4c67-4614-9ff6-e122b5d3c55e","Type":"ContainerDied","Data":"a1f87ba8738cc681d1f3a9cc3c95a3a66859c9672b4f9af7322cbdbbb1fd651a"} Jul 14 22:19:57.920561 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:57.920411 2562 scope.go:117] "RemoveContainer" containerID="a1f87ba8738cc681d1f3a9cc3c95a3a66859c9672b4f9af7322cbdbbb1fd651a" Jul 14 22:19:58.927921 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:58.927886 2562 generic.go:356] "Generic (PLEG): container finished" podID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerID="38d5a5fbed99f0b2c0f80d03bc120691826b7ff0d729c3d297fa296410069ca6" exitCode=7 Jul 14 22:19:58.928375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:58.927981 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" event={"ID":"fa8d1246-4c67-4614-9ff6-e122b5d3c55e","Type":"ContainerDied","Data":"38d5a5fbed99f0b2c0f80d03bc120691826b7ff0d729c3d297fa296410069ca6"} Jul 14 22:19:58.928375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:58.928068 2562 scope.go:117] "RemoveContainer" containerID="a1f87ba8738cc681d1f3a9cc3c95a3a66859c9672b4f9af7322cbdbbb1fd651a" Jul 14 22:19:58.928375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:58.928314 2562 scope.go:117] "RemoveContainer" containerID="38d5a5fbed99f0b2c0f80d03bc120691826b7ff0d729c3d297fa296410069ca6" Jul 14 22:19:58.928629 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:19:58.928601 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf_odh-ai-gateway-infra(fa8d1246-4c67-4614-9ff6-e122b5d3c55e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" Jul 14 22:19:59.935411 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:19:59.935384 2562 scope.go:117] "RemoveContainer" containerID="38d5a5fbed99f0b2c0f80d03bc120691826b7ff0d729c3d297fa296410069ca6" Jul 14 22:19:59.935793 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:19:59.935628 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf_odh-ai-gateway-infra(fa8d1246-4c67-4614-9ff6-e122b5d3c55e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" Jul 14 22:20:02.955663 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:02.955575 2562 generic.go:356] "Generic (PLEG): container finished" podID="cf415728-e45e-4556-8290-eccbd3fba786" containerID="362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0" exitCode=0 Jul 14 22:20:02.955663 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:02.955628 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" event={"ID":"cf415728-e45e-4556-8290-eccbd3fba786","Type":"ContainerDied","Data":"362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0"} Jul 14 22:20:02.956138 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:02.955995 2562 scope.go:117] "RemoveContainer" containerID="362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0" Jul 14 22:20:02.957043 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:02.956992 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:20:02.957107 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:02.957088 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:20:02.957143 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:02.957132 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:20:03.963971 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:03.963932 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" event={"ID":"cf415728-e45e-4556-8290-eccbd3fba786","Type":"ContainerStarted","Data":"d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807"} Jul 14 22:20:04.351939 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:04.351898 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:20:04.353162 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:04.353133 2562 patch_prober.go:28] interesting pod/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 14 22:20:04.353290 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:04.353184 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 14 22:20:05.351551 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:05.351510 2562 patch_prober.go:28] interesting pod/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 14 22:20:05.351970 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:05.351582 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 14 22:20:06.351842 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:06.351805 2562 patch_prober.go:28] interesting pod/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 14 22:20:06.352356 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:06.351893 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 14 22:20:06.879235 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:06.879203 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:20:07.352000 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:07.351958 2562 patch_prober.go:28] interesting pod/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 14 22:20:07.352426 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:07.352045 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 14 22:20:08.355615 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:08.355583 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:20:08.356630 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:08.356614 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:20:11.804725 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:11.804693 2562 scope.go:117] "RemoveContainer" containerID="38d5a5fbed99f0b2c0f80d03bc120691826b7ff0d729c3d297fa296410069ca6" Jul 14 22:20:13.014889 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:13.014852 2562 generic.go:356] "Generic (PLEG): container finished" podID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerID="3450c6c8a3b9eeb27409a38c75d51e5d0371cb3711e38e0a5e38ae4cfa18c049" exitCode=0 Jul 14 22:20:13.015345 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:13.014925 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" event={"ID":"fa8d1246-4c67-4614-9ff6-e122b5d3c55e","Type":"ContainerDied","Data":"3450c6c8a3b9eeb27409a38c75d51e5d0371cb3711e38e0a5e38ae4cfa18c049"} Jul 14 22:20:13.015345 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:13.014986 2562 scope.go:117] "RemoveContainer" containerID="38d5a5fbed99f0b2c0f80d03bc120691826b7ff0d729c3d297fa296410069ca6" Jul 14 22:20:14.044070 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:14.044010 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf"] Jul 14 22:20:14.159246 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:14.159217 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" Jul 14 22:20:14.194815 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:14.194785 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8d1246-4c67-4614-9ff6-e122b5d3c55e-maas-api-service-ca\") pod \"fa8d1246-4c67-4614-9ff6-e122b5d3c55e\" (UID: \"fa8d1246-4c67-4614-9ff6-e122b5d3c55e\") " Jul 14 22:20:14.195209 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:14.195183 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8d1246-4c67-4614-9ff6-e122b5d3c55e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa8d1246-4c67-4614-9ff6-e122b5d3c55e" (UID: "fa8d1246-4c67-4614-9ff6-e122b5d3c55e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:20:14.295947 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:14.295861 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8d1246-4c67-4614-9ff6-e122b5d3c55e-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:20:15.030522 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:15.030496 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" Jul 14 22:20:15.030702 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:15.030500 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf" event={"ID":"fa8d1246-4c67-4614-9ff6-e122b5d3c55e","Type":"ContainerDied","Data":"f89730df82eeae025044bcb217cd98df6e2ffac13a73d382b76ca719f9d57799"} Jul 14 22:20:15.030702 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:15.030630 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f89730df82eeae025044bcb217cd98df6e2ffac13a73d382b76ca719f9d57799" Jul 14 22:20:15.063581 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:15.063551 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf"] Jul 14 22:20:15.067362 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:15.067295 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-tfqcf"] Jul 14 22:20:16.810337 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:16.810299 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" path="/var/lib/kubelet/pods/fa8d1246-4c67-4614-9ff6-e122b5d3c55e/volumes" Jul 14 22:20:46.358881 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.358842 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm"] Jul 14 22:20:46.359375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.359222 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerName="revoke-keys" Jul 14 22:20:46.359375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.359235 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerName="revoke-keys" Jul 14 22:20:46.359375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.359245 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerName="revoke-keys" Jul 14 22:20:46.359375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.359251 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerName="revoke-keys" Jul 14 22:20:46.359375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.359314 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerName="revoke-keys" Jul 14 22:20:46.359375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.359324 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerName="revoke-keys" Jul 14 22:20:46.359375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.359330 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerName="revoke-keys" Jul 14 22:20:46.361239 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.361222 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" Jul 14 22:20:46.364207 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.364185 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:20:46.364437 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.364421 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:20:46.375254 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.375230 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm"] Jul 14 22:20:46.391166 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.391135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f988bbd-539e-43c1-af4b-2252ea15c184-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c29be32a-p82sm\" (UID: \"4f988bbd-539e-43c1-af4b-2252ea15c184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" Jul 14 22:20:46.491640 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.491602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f988bbd-539e-43c1-af4b-2252ea15c184-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c29be32a-p82sm\" (UID: \"4f988bbd-539e-43c1-af4b-2252ea15c184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" Jul 14 22:20:46.492257 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.492238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f988bbd-539e-43c1-af4b-2252ea15c184-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c29be32a-p82sm\" (UID: \"4f988bbd-539e-43c1-af4b-2252ea15c184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" Jul 14 22:20:46.671428 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.671335 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" Jul 14 22:20:46.810413 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:46.810388 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm"] Jul 14 22:20:46.811776 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:20:46.811749 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f988bbd_539e_43c1_af4b_2252ea15c184.slice/crio-f92dc709292351acbf4dbfda2dff71e910f4a458c6d2053b9a2e95fbfa314cf6 WatchSource:0}: Error finding container f92dc709292351acbf4dbfda2dff71e910f4a458c6d2053b9a2e95fbfa314cf6: Status 404 returned error can't find the container with id f92dc709292351acbf4dbfda2dff71e910f4a458c6d2053b9a2e95fbfa314cf6 Jul 14 22:20:47.214206 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:47.214163 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" event={"ID":"4f988bbd-539e-43c1-af4b-2252ea15c184","Type":"ContainerStarted","Data":"bcf4f6d060879a381a804f245b69f827390da4caf54ccd8f56e2ff9b18b52f74"} Jul 14 22:20:47.214405 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:47.214251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" event={"ID":"4f988bbd-539e-43c1-af4b-2252ea15c184","Type":"ContainerStarted","Data":"f92dc709292351acbf4dbfda2dff71e910f4a458c6d2053b9a2e95fbfa314cf6"} Jul 14 22:20:47.232752 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:47.232698 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" podStartSLOduration=1.232682578 podStartE2EDuration="1.232682578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:20:47.231141025 +0000 UTC m=+1782.080611067" watchObservedRunningTime="2026-07-14 22:20:47.232682578 +0000 UTC m=+1782.082152616" Jul 14 22:20:48.222437 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:48.222402 2562 generic.go:356] "Generic (PLEG): container finished" podID="4f988bbd-539e-43c1-af4b-2252ea15c184" containerID="bcf4f6d060879a381a804f245b69f827390da4caf54ccd8f56e2ff9b18b52f74" exitCode=0 Jul 14 22:20:48.222889 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:48.222445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" event={"ID":"4f988bbd-539e-43c1-af4b-2252ea15c184","Type":"ContainerDied","Data":"bcf4f6d060879a381a804f245b69f827390da4caf54ccd8f56e2ff9b18b52f74"} Jul 14 22:20:49.367334 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:49.367308 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" Jul 14 22:20:49.415111 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:49.415076 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f988bbd-539e-43c1-af4b-2252ea15c184-maas-api-service-ca\") pod \"4f988bbd-539e-43c1-af4b-2252ea15c184\" (UID: \"4f988bbd-539e-43c1-af4b-2252ea15c184\") " Jul 14 22:20:49.415448 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:49.415424 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f988bbd-539e-43c1-af4b-2252ea15c184-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f988bbd-539e-43c1-af4b-2252ea15c184" (UID: "4f988bbd-539e-43c1-af4b-2252ea15c184"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:20:49.516421 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:49.516318 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f988bbd-539e-43c1-af4b-2252ea15c184-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:20:50.234418 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:50.234390 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" Jul 14 22:20:50.234418 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:50.234409 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c29be32a-p82sm" event={"ID":"4f988bbd-539e-43c1-af4b-2252ea15c184","Type":"ContainerDied","Data":"f92dc709292351acbf4dbfda2dff71e910f4a458c6d2053b9a2e95fbfa314cf6"} Jul 14 22:20:50.234632 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:20:50.234451 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f92dc709292351acbf4dbfda2dff71e910f4a458c6d2053b9a2e95fbfa314cf6" Jul 14 22:21:12.428454 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.428423 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc"] Jul 14 22:21:12.428887 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.428658 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" containerID="cri-o://d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807" gracePeriod=30 Jul 14 22:21:12.723320 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.723240 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh"] Jul 14 22:21:12.723806 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.723790 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerName="revoke-keys" Jul 14 22:21:12.723848 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.723809 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8d1246-4c67-4614-9ff6-e122b5d3c55e" containerName="revoke-keys" Jul 14 22:21:12.723848 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.723827 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f988bbd-539e-43c1-af4b-2252ea15c184" containerName="revoke-keys" Jul 14 22:21:12.723848 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.723832 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f988bbd-539e-43c1-af4b-2252ea15c184" containerName="revoke-keys" Jul 14 22:21:12.723946 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.723888 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f988bbd-539e-43c1-af4b-2252ea15c184" containerName="revoke-keys" Jul 14 22:21:12.725787 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.725767 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.729877 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.729853 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6b2eb522-openshift-default-dockercfg-jl672\"" Jul 14 22:21:12.741671 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.741648 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh"] Jul 14 22:21:12.824277 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.824241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-podinfo\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.824624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.824285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-token\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.824624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.824387 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istiod-ca-cert\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.824624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.824455 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-269kl\" (UniqueName: \"kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-kube-api-access-269kl\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.824624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.824506 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-credential-socket\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.824624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.824531 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-envoy\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.824624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.824567 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-certs\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.824959 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.824651 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-socket\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.824959 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.824691 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-data\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.925550 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.925496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-socket\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.925550 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.925549 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-data\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.925815 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.925594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-podinfo\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.925815 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.925611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-token\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.925815 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.925628 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istiod-ca-cert\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.925815 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.925675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-269kl\" (UniqueName: \"kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-kube-api-access-269kl\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.925815 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.925722 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-credential-socket\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.925815 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.925744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-envoy\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.925815 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.925780 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-certs\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.926222 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.925981 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-socket\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.926222 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.926093 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-data\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.926330 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.926233 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-credential-socket\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.926408 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.926376 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-certs\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.926461 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.926447 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istiod-ca-cert\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.928070 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.928048 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-podinfo\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.928166 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.928090 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-envoy\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.935169 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.935143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-token\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:12.935259 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:12.935222 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-269kl\" (UniqueName: \"kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-kube-api-access-269kl\") pod \"e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:13.038161 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:13.038133 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:13.187107 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:13.187077 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh"] Jul 14 22:21:13.188106 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:21:13.188078 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d4c5dc_950c_4de1_bd8d_2d8fcb7c2c39.slice/crio-afe1684190b68973c75670bed5aee0c47c9ff5eed1e859dbe64a4735ddc1accd WatchSource:0}: Error finding container afe1684190b68973c75670bed5aee0c47c9ff5eed1e859dbe64a4735ddc1accd: Status 404 returned error can't find the container with id afe1684190b68973c75670bed5aee0c47c9ff5eed1e859dbe64a4735ddc1accd Jul 14 22:21:13.190630 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:13.190600 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:21:13.190689 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:13.190663 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:21:13.190734 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:13.190705 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:21:13.361093 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:13.361065 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" event={"ID":"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39","Type":"ContainerStarted","Data":"e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5"} Jul 14 22:21:13.361232 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:13.361124 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" event={"ID":"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39","Type":"ContainerStarted","Data":"afe1684190b68973c75670bed5aee0c47c9ff5eed1e859dbe64a4735ddc1accd"} Jul 14 22:21:13.385010 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:13.384954 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" podStartSLOduration=1.384937289 podStartE2EDuration="1.384937289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:21:13.381932206 +0000 UTC m=+1808.231402263" watchObservedRunningTime="2026-07-14 22:21:13.384937289 +0000 UTC m=+1808.234407328" Jul 14 22:21:14.038489 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:14.038449 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:14.043727 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:14.043696 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:14.366593 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:14.366508 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:14.367466 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:14.367444 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:17.715742 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.715666 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:21:17.876326 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876285 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-data\") pod \"cf415728-e45e-4556-8290-eccbd3fba786\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " Jul 14 22:21:17.876518 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876346 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-socket\") pod \"cf415728-e45e-4556-8290-eccbd3fba786\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " Jul 14 22:21:17.876518 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876388 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-istio-token\") pod \"cf415728-e45e-4556-8290-eccbd3fba786\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " Jul 14 22:21:17.876518 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876426 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-credential-socket\") pod \"cf415728-e45e-4556-8290-eccbd3fba786\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " Jul 14 22:21:17.876518 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876469 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvxl4\" (UniqueName: \"kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-kube-api-access-cvxl4\") pod \"cf415728-e45e-4556-8290-eccbd3fba786\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " Jul 14 22:21:17.876518 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876516 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf415728-e45e-4556-8290-eccbd3fba786-istio-podinfo\") pod \"cf415728-e45e-4556-8290-eccbd3fba786\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " Jul 14 22:21:17.876781 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876543 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-certs\") pod \"cf415728-e45e-4556-8290-eccbd3fba786\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " Jul 14 22:21:17.876781 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876570 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-envoy\") pod \"cf415728-e45e-4556-8290-eccbd3fba786\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " Jul 14 22:21:17.876781 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876600 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf415728-e45e-4556-8290-eccbd3fba786-istiod-ca-cert\") pod \"cf415728-e45e-4556-8290-eccbd3fba786\" (UID: \"cf415728-e45e-4556-8290-eccbd3fba786\") " Jul 14 22:21:17.876781 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876629 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf415728-e45e-4556-8290-eccbd3fba786" (UID: "cf415728-e45e-4556-8290-eccbd3fba786"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:21:17.876781 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876756 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf415728-e45e-4556-8290-eccbd3fba786" (UID: "cf415728-e45e-4556-8290-eccbd3fba786"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:21:17.877075 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876871 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf415728-e45e-4556-8290-eccbd3fba786" (UID: "cf415728-e45e-4556-8290-eccbd3fba786"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:21:17.877075 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876954 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:21:17.877075 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.876994 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:21:17.877075 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.877039 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf415728-e45e-4556-8290-eccbd3fba786-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf415728-e45e-4556-8290-eccbd3fba786" (UID: "cf415728-e45e-4556-8290-eccbd3fba786"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:21:17.877319 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.877300 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf415728-e45e-4556-8290-eccbd3fba786" (UID: "cf415728-e45e-4556-8290-eccbd3fba786"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:21:17.879289 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.879264 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf415728-e45e-4556-8290-eccbd3fba786" (UID: "cf415728-e45e-4556-8290-eccbd3fba786"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:21:17.879395 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.879296 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf415728-e45e-4556-8290-eccbd3fba786" (UID: "cf415728-e45e-4556-8290-eccbd3fba786"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:21:17.879395 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.879304 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-kube-api-access-cvxl4" (OuterVolumeSpecName: "kube-api-access-cvxl4") pod "cf415728-e45e-4556-8290-eccbd3fba786" (UID: "cf415728-e45e-4556-8290-eccbd3fba786"). InnerVolumeSpecName "kube-api-access-cvxl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:21:17.879469 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.879386 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf415728-e45e-4556-8290-eccbd3fba786-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf415728-e45e-4556-8290-eccbd3fba786" (UID: "cf415728-e45e-4556-8290-eccbd3fba786"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:21:17.977820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.977713 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf415728-e45e-4556-8290-eccbd3fba786-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:21:17.977820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.977756 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:21:17.977820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.977765 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:21:17.977820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.977774 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf415728-e45e-4556-8290-eccbd3fba786-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:21:17.977820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.977782 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf415728-e45e-4556-8290-eccbd3fba786-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:21:17.977820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.977790 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:21:17.977820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:17.977798 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvxl4\" (UniqueName: \"kubernetes.io/projected/cf415728-e45e-4556-8290-eccbd3fba786-kube-api-access-cvxl4\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:21:18.267574 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.267483 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2"] Jul 14 22:21:18.267851 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.267839 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" Jul 14 22:21:18.267902 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.267852 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" Jul 14 22:21:18.267902 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.267862 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" Jul 14 22:21:18.267902 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.267867 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" Jul 14 22:21:18.267991 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.267940 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" Jul 14 22:21:18.272221 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.272204 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" Jul 14 22:21:18.275207 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.275180 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:21:18.276057 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.275844 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:21:18.278312 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.278286 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2"] Jul 14 22:21:18.381555 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.381518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01a09dc-aa66-46f0-842c-7b822c69ab5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2\" (UID: \"f01a09dc-aa66-46f0-842c-7b822c69ab5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" Jul 14 22:21:18.402047 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.397706 2562 generic.go:356] "Generic (PLEG): container finished" podID="cf415728-e45e-4556-8290-eccbd3fba786" containerID="d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807" exitCode=0 Jul 14 22:21:18.402047 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.397905 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" event={"ID":"cf415728-e45e-4556-8290-eccbd3fba786","Type":"ContainerDied","Data":"d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807"} Jul 14 22:21:18.402047 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.397962 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" event={"ID":"cf415728-e45e-4556-8290-eccbd3fba786","Type":"ContainerDied","Data":"bd3c0627eb76342163970ec65cf4e965eec04c7e882669299d62ff63f656c9ee"} Jul 14 22:21:18.402047 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.397994 2562 scope.go:117] "RemoveContainer" containerID="d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807" Jul 14 22:21:18.402047 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.398193 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc" Jul 14 22:21:18.421841 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.421816 2562 scope.go:117] "RemoveContainer" containerID="362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0" Jul 14 22:21:18.430637 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.430605 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc"] Jul 14 22:21:18.431664 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.431637 2562 scope.go:117] "RemoveContainer" containerID="d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807" Jul 14 22:21:18.432017 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:21:18.431989 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807\": container with ID starting with d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807 not found: ID does not exist" containerID="d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807" Jul 14 22:21:18.432142 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.432038 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807"} err="failed to get container status \"d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807\": rpc error: code = NotFound desc = could not find container \"d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807\": container with ID starting with d4dc6a70956079f76acbc47a7898348376023f2b21a6b237242f1dfb5d27b807 not found: ID does not exist" Jul 14 22:21:18.432142 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.432064 2562 scope.go:117] "RemoveContainer" containerID="362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0" Jul 14 22:21:18.432371 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:21:18.432349 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0\": container with ID starting with 362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0 not found: ID does not exist" containerID="362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0" Jul 14 22:21:18.432449 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.432386 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0"} err="failed to get container status \"362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0\": rpc error: code = NotFound desc = could not find container \"362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0\": container with ID starting with 362329399e47a4dff1722abe51556ea47dc6ffeba681c2f0d3843f1064f6cfd0 not found: ID does not exist" Jul 14 22:21:18.434001 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.433981 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c29be32a-openshift-default-587c746bd5-6fjrc"] Jul 14 22:21:18.481949 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.481911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01a09dc-aa66-46f0-842c-7b822c69ab5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2\" (UID: \"f01a09dc-aa66-46f0-842c-7b822c69ab5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" Jul 14 22:21:18.482872 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.482816 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01a09dc-aa66-46f0-842c-7b822c69ab5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2\" (UID: \"f01a09dc-aa66-46f0-842c-7b822c69ab5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" Jul 14 22:21:18.602048 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.600167 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" Jul 14 22:21:18.765683 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.765653 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2"] Jul 14 22:21:18.767578 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:21:18.767547 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01a09dc_aa66_46f0_842c_7b822c69ab5f.slice/crio-a353d5aa4ec9dc48a5038541cd20e4707a8f32a90d92a2752419db676fa54e42 WatchSource:0}: Error finding container a353d5aa4ec9dc48a5038541cd20e4707a8f32a90d92a2752419db676fa54e42: Status 404 returned error can't find the container with id a353d5aa4ec9dc48a5038541cd20e4707a8f32a90d92a2752419db676fa54e42 Jul 14 22:21:18.813629 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:18.813598 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf415728-e45e-4556-8290-eccbd3fba786" path="/var/lib/kubelet/pods/cf415728-e45e-4556-8290-eccbd3fba786/volumes" Jul 14 22:21:19.410730 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:19.410681 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" event={"ID":"f01a09dc-aa66-46f0-842c-7b822c69ab5f","Type":"ContainerStarted","Data":"d7de08d92d0d6da072603a9bea91df7e733cd5eb37d7847c347463669a1c09f2"} Jul 14 22:21:19.410942 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:19.410750 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" event={"ID":"f01a09dc-aa66-46f0-842c-7b822c69ab5f","Type":"ContainerStarted","Data":"a353d5aa4ec9dc48a5038541cd20e4707a8f32a90d92a2752419db676fa54e42"} Jul 14 22:21:19.428485 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:19.428434 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" podStartSLOduration=1.428418468 podStartE2EDuration="1.428418468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:21:19.427382202 +0000 UTC m=+1814.276852243" watchObservedRunningTime="2026-07-14 22:21:19.428418468 +0000 UTC m=+1814.277888507" Jul 14 22:21:20.418599 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:20.418563 2562 generic.go:356] "Generic (PLEG): container finished" podID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerID="d7de08d92d0d6da072603a9bea91df7e733cd5eb37d7847c347463669a1c09f2" exitCode=7 Jul 14 22:21:20.419004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:20.418659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" event={"ID":"f01a09dc-aa66-46f0-842c-7b822c69ab5f","Type":"ContainerDied","Data":"d7de08d92d0d6da072603a9bea91df7e733cd5eb37d7847c347463669a1c09f2"} Jul 14 22:21:20.419004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:20.418860 2562 scope.go:117] "RemoveContainer" containerID="d7de08d92d0d6da072603a9bea91df7e733cd5eb37d7847c347463669a1c09f2" Jul 14 22:21:21.426866 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:21.426835 2562 generic.go:356] "Generic (PLEG): container finished" podID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerID="240b9807c9294a893bda9c2031cff4bae77c040ef1c22b2c968be8f532774628" exitCode=7 Jul 14 22:21:21.427292 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:21.426900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" event={"ID":"f01a09dc-aa66-46f0-842c-7b822c69ab5f","Type":"ContainerDied","Data":"240b9807c9294a893bda9c2031cff4bae77c040ef1c22b2c968be8f532774628"} Jul 14 22:21:21.427292 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:21.426967 2562 scope.go:117] "RemoveContainer" containerID="d7de08d92d0d6da072603a9bea91df7e733cd5eb37d7847c347463669a1c09f2" Jul 14 22:21:21.427292 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:21.427166 2562 scope.go:117] "RemoveContainer" containerID="240b9807c9294a893bda9c2031cff4bae77c040ef1c22b2c968be8f532774628" Jul 14 22:21:21.427499 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:21:21.427477 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2_odh-ai-gateway-infra(f01a09dc-aa66-46f0-842c-7b822c69ab5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" Jul 14 22:21:22.434844 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:22.434809 2562 scope.go:117] "RemoveContainer" containerID="240b9807c9294a893bda9c2031cff4bae77c040ef1c22b2c968be8f532774628" Jul 14 22:21:22.435243 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:21:22.435053 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2_odh-ai-gateway-infra(f01a09dc-aa66-46f0-842c-7b822c69ab5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" Jul 14 22:21:25.452039 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:25.452001 2562 generic.go:356] "Generic (PLEG): container finished" podID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerID="e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5" exitCode=0 Jul 14 22:21:25.452533 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:25.452063 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" event={"ID":"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39","Type":"ContainerDied","Data":"e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5"} Jul 14 22:21:25.452533 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:25.452436 2562 scope.go:117] "RemoveContainer" containerID="e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5" Jul 14 22:21:25.453389 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:25.453360 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:21:25.453447 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:25.453426 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:21:25.453486 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:25.453457 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:21:26.460316 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:26.460282 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" event={"ID":"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39","Type":"ContainerStarted","Data":"a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307"} Jul 14 22:21:27.038577 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:27.038537 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:27.039552 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:27.039527 2562 patch_prober.go:28] interesting pod/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 22:21:27.039617 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:27.039580 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 22:21:28.039310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:28.039272 2562 patch_prober.go:28] interesting pod/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 22:21:28.039730 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:28.039335 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 22:21:29.039597 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:29.039554 2562 patch_prober.go:28] interesting pod/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 22:21:29.040005 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:29.039613 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 22:21:29.367464 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:29.367372 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:30.038901 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:30.038864 2562 patch_prober.go:28] interesting pod/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 22:21:30.039100 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:30.038935 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 22:21:31.042820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:31.042788 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:31.043861 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:31.043837 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:21:36.810780 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:36.810748 2562 scope.go:117] "RemoveContainer" containerID="240b9807c9294a893bda9c2031cff4bae77c040ef1c22b2c968be8f532774628" Jul 14 22:21:37.522110 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:37.522068 2562 generic.go:356] "Generic (PLEG): container finished" podID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerID="29104b53e718932c2a48237cb6d20b08cf5194a2ae6f79dfd8063fa35e3377fd" exitCode=0 Jul 14 22:21:37.522282 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:37.522132 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" event={"ID":"f01a09dc-aa66-46f0-842c-7b822c69ab5f","Type":"ContainerDied","Data":"29104b53e718932c2a48237cb6d20b08cf5194a2ae6f79dfd8063fa35e3377fd"} Jul 14 22:21:37.522282 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:37.522190 2562 scope.go:117] "RemoveContainer" containerID="240b9807c9294a893bda9c2031cff4bae77c040ef1c22b2c968be8f532774628" Jul 14 22:21:38.556179 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:38.556144 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2"] Jul 14 22:21:38.662852 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:38.662827 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" Jul 14 22:21:38.695201 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:38.695173 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01a09dc-aa66-46f0-842c-7b822c69ab5f-maas-api-service-ca\") pod \"f01a09dc-aa66-46f0-842c-7b822c69ab5f\" (UID: \"f01a09dc-aa66-46f0-842c-7b822c69ab5f\") " Jul 14 22:21:38.695515 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:38.695494 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01a09dc-aa66-46f0-842c-7b822c69ab5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f01a09dc-aa66-46f0-842c-7b822c69ab5f" (UID: "f01a09dc-aa66-46f0-842c-7b822c69ab5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:21:38.796534 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:38.796441 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01a09dc-aa66-46f0-842c-7b822c69ab5f-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:21:39.536156 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:39.536126 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" Jul 14 22:21:39.536156 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:39.536147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2" event={"ID":"f01a09dc-aa66-46f0-842c-7b822c69ab5f","Type":"ContainerDied","Data":"a353d5aa4ec9dc48a5038541cd20e4707a8f32a90d92a2752419db676fa54e42"} Jul 14 22:21:39.536385 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:39.536191 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a353d5aa4ec9dc48a5038541cd20e4707a8f32a90d92a2752419db676fa54e42" Jul 14 22:21:39.575510 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:39.575473 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2"] Jul 14 22:21:39.581259 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:39.581232 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-bvmn2"] Jul 14 22:21:40.810492 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:21:40.810456 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" path="/var/lib/kubelet/pods/f01a09dc-aa66-46f0-842c-7b822c69ab5f/volumes" Jul 14 22:22:18.318221 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.318185 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp"] Jul 14 22:22:18.318666 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.318543 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerName="revoke-keys" Jul 14 22:22:18.318666 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.318554 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerName="revoke-keys" Jul 14 22:22:18.318666 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.318567 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerName="revoke-keys" Jul 14 22:22:18.318666 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.318572 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerName="revoke-keys" Jul 14 22:22:18.318666 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.318642 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerName="revoke-keys" Jul 14 22:22:18.318666 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.318649 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerName="revoke-keys" Jul 14 22:22:18.318666 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.318656 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf415728-e45e-4556-8290-eccbd3fba786" containerName="istio-proxy" Jul 14 22:22:18.321573 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.321558 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp" Jul 14 22:22:18.324492 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.324455 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:22:18.324664 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.324546 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:22:18.328676 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.328643 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp"] Jul 14 22:22:18.346915 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.346892 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f085c840-e51a-4239-be21-6c7c80e0ac84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp\" (UID: \"f085c840-e51a-4239-be21-6c7c80e0ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp" Jul 14 22:22:18.447775 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.447738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f085c840-e51a-4239-be21-6c7c80e0ac84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp\" (UID: \"f085c840-e51a-4239-be21-6c7c80e0ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp" Jul 14 22:22:18.448387 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.448364 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f085c840-e51a-4239-be21-6c7c80e0ac84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp\" (UID: \"f085c840-e51a-4239-be21-6c7c80e0ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp" Jul 14 22:22:18.634158 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.634069 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp" Jul 14 22:22:18.770690 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:18.770663 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp"] Jul 14 22:22:18.772218 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:22:18.772190 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf085c840_e51a_4239_be21_6c7c80e0ac84.slice/crio-5ff7a16e068e5d184c7d5c0a5aeb12752b452a19cdf66b9fea7ac9c737b76a68 WatchSource:0}: Error finding container 5ff7a16e068e5d184c7d5c0a5aeb12752b452a19cdf66b9fea7ac9c737b76a68: Status 404 returned error can't find the container with id 5ff7a16e068e5d184c7d5c0a5aeb12752b452a19cdf66b9fea7ac9c737b76a68 Jul 14 22:22:19.759510 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:19.759415 2562 generic.go:356] "Generic (PLEG): container finished" podID="f085c840-e51a-4239-be21-6c7c80e0ac84" containerID="89a4add819a654ecad495e3b70a32ddf4e1f5b9c7a52d405fe6b5b320f859eee" exitCode=0 Jul 14 22:22:19.759881 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:19.759503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp" event={"ID":"f085c840-e51a-4239-be21-6c7c80e0ac84","Type":"ContainerDied","Data":"89a4add819a654ecad495e3b70a32ddf4e1f5b9c7a52d405fe6b5b320f859eee"} Jul 14 22:22:19.759881 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:19.759552 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp" event={"ID":"f085c840-e51a-4239-be21-6c7c80e0ac84","Type":"ContainerStarted","Data":"5ff7a16e068e5d184c7d5c0a5aeb12752b452a19cdf66b9fea7ac9c737b76a68"} Jul 14 22:22:20.897627 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:20.897600 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp" Jul 14 22:22:20.968955 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:20.968913 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f085c840-e51a-4239-be21-6c7c80e0ac84-maas-api-service-ca\") pod \"f085c840-e51a-4239-be21-6c7c80e0ac84\" (UID: \"f085c840-e51a-4239-be21-6c7c80e0ac84\") " Jul 14 22:22:20.969316 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:20.969292 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f085c840-e51a-4239-be21-6c7c80e0ac84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f085c840-e51a-4239-be21-6c7c80e0ac84" (UID: "f085c840-e51a-4239-be21-6c7c80e0ac84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:22:21.070510 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:21.070464 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f085c840-e51a-4239-be21-6c7c80e0ac84-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:22:21.772406 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:21.772372 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp" Jul 14 22:22:21.772583 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:21.772419 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp" event={"ID":"f085c840-e51a-4239-be21-6c7c80e0ac84","Type":"ContainerDied","Data":"5ff7a16e068e5d184c7d5c0a5aeb12752b452a19cdf66b9fea7ac9c737b76a68"} Jul 14 22:22:21.772583 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:21.772467 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff7a16e068e5d184c7d5c0a5aeb12752b452a19cdf66b9fea7ac9c737b76a68" Jul 14 22:22:43.587380 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:43.587294 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh"] Jul 14 22:22:43.587880 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:43.587524 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" containerID="cri-o://a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307" gracePeriod=30 Jul 14 22:22:46.045857 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:46.045810 2562 patch_prober.go:28] interesting pod/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:22:46.046329 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:46.045898 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:22:48.854526 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.854462 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:22:48.925816 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.925779 2562 generic.go:356] "Generic (PLEG): container finished" podID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerID="a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307" exitCode=0 Jul 14 22:22:48.925982 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.925848 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" Jul 14 22:22:48.925982 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.925856 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" event={"ID":"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39","Type":"ContainerDied","Data":"a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307"} Jul 14 22:22:48.925982 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.925938 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh" event={"ID":"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39","Type":"ContainerDied","Data":"afe1684190b68973c75670bed5aee0c47c9ff5eed1e859dbe64a4735ddc1accd"} Jul 14 22:22:48.925982 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.925960 2562 scope.go:117] "RemoveContainer" containerID="a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307" Jul 14 22:22:48.935603 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.935574 2562 scope.go:117] "RemoveContainer" containerID="e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5" Jul 14 22:22:48.944556 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.944535 2562 scope.go:117] "RemoveContainer" containerID="a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307" Jul 14 22:22:48.944831 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:22:48.944811 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307\": container with ID starting with a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307 not found: ID does not exist" containerID="a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307" Jul 14 22:22:48.944883 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.944841 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307"} err="failed to get container status \"a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307\": rpc error: code = NotFound desc = could not find container \"a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307\": container with ID starting with a0eaff177a96bb6a70ee0057c4b9d1332459e40cd24307ca3892ef9429fdc307 not found: ID does not exist" Jul 14 22:22:48.944883 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.944860 2562 scope.go:117] "RemoveContainer" containerID="e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5" Jul 14 22:22:48.945170 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:22:48.945151 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5\": container with ID starting with e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5 not found: ID does not exist" containerID="e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5" Jul 14 22:22:48.945249 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:48.945175 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5"} err="failed to get container status \"e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5\": rpc error: code = NotFound desc = could not find container \"e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5\": container with ID starting with e25c5aaaeb0c5cb0f42b931959f048ed94f72c5ac22ad70aaf2762f8b21f4ff5 not found: ID does not exist" Jul 14 22:22:49.028333 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028302 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-269kl\" (UniqueName: \"kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-kube-api-access-269kl\") pod \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " Jul 14 22:22:49.028540 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028362 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-socket\") pod \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " Jul 14 22:22:49.028540 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028413 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-envoy\") pod \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " Jul 14 22:22:49.028540 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028458 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-podinfo\") pod \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " Jul 14 22:22:49.028540 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028487 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-token\") pod \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " Jul 14 22:22:49.028540 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028528 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-credential-socket\") pod \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " Jul 14 22:22:49.028820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028561 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-certs\") pod \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " Jul 14 22:22:49.028820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028700 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" (UID: "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:22:49.028820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028743 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-data\") pod \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " Jul 14 22:22:49.028820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028785 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" (UID: "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:22:49.029072 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028824 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istiod-ca-cert\") pod \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\" (UID: \"98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39\") " Jul 14 22:22:49.029072 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.028860 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" (UID: "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:22:49.029197 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.029091 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" (UID: "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:22:49.029253 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.029208 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:22:49.029253 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.029229 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:22:49.029253 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.029242 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:22:49.029415 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.029254 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:22:49.029736 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.029703 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-data" (OuterVolumeSpecName: "istio-data") pod "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" (UID: "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:22:49.030770 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.030743 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" (UID: "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:22:49.030872 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.030783 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-kube-api-access-269kl" (OuterVolumeSpecName: "kube-api-access-269kl") pod "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" (UID: "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39"). InnerVolumeSpecName "kube-api-access-269kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:22:49.031512 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.031482 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-token" (OuterVolumeSpecName: "istio-token") pod "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" (UID: "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:22:49.031614 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.031521 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" (UID: "98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:22:49.130088 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.129972 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:22:49.130088 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.130004 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:22:49.130088 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.130014 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:22:49.130088 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.130049 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:22:49.130088 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.130058 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-269kl\" (UniqueName: \"kubernetes.io/projected/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39-kube-api-access-269kl\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:22:49.249581 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.249540 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh"] Jul 14 22:22:49.253909 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:49.253876 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6b2eb522-openshift-default-66849fb867-jvwdh"] Jul 14 22:22:50.810568 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:22:50.810534 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" path="/var/lib/kubelet/pods/98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39/volumes" Jul 14 22:24:06.954540 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:06.954458 2562 scope.go:117] "RemoveContainer" containerID="0f41c99f44119770f30a7e624696e4dc678f88f8d3e014136782004dd6d9dcbd" Jul 14 22:24:20.175797 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.175763 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68"] Jul 14 22:24:20.176295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.176177 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerName="revoke-keys" Jul 14 22:24:20.176295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.176192 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerName="revoke-keys" Jul 14 22:24:20.176295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.176202 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" Jul 14 22:24:20.176295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.176209 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" Jul 14 22:24:20.176295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.176220 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f085c840-e51a-4239-be21-6c7c80e0ac84" containerName="revoke-keys" Jul 14 22:24:20.176295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.176226 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f085c840-e51a-4239-be21-6c7c80e0ac84" containerName="revoke-keys" Jul 14 22:24:20.176295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.176286 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f085c840-e51a-4239-be21-6c7c80e0ac84" containerName="revoke-keys" Jul 14 22:24:20.176295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.176296 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" Jul 14 22:24:20.176592 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.176303 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f01a09dc-aa66-46f0-842c-7b822c69ab5f" containerName="revoke-keys" Jul 14 22:24:20.179409 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.179392 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.182375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.182354 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c488720f-openshift-default-dockercfg-84qdx\"" Jul 14 22:24:20.203167 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.203137 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68"] Jul 14 22:24:20.326602 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.326569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2578654c-9910-41d7-9015-c1bc5ac43911-istio-podinfo\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.326807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.326619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-envoy\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.326807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.326642 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njntm\" (UniqueName: \"kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-kube-api-access-njntm\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.326807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.326711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-socket\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.326807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.326731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-credential-socket\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.326807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.326754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-certs\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.326807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.326781 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2578654c-9910-41d7-9015-c1bc5ac43911-istiod-ca-cert\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.326807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.326798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-data\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.327090 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.326818 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-istio-token\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.427966 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.427873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2578654c-9910-41d7-9015-c1bc5ac43911-istio-podinfo\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.427966 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.427920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-envoy\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.427966 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.427941 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njntm\" (UniqueName: \"kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-kube-api-access-njntm\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428276 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.427970 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-socket\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428276 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.428009 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-credential-socket\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428276 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.428070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-certs\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428276 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.428102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2578654c-9910-41d7-9015-c1bc5ac43911-istiod-ca-cert\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428276 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.428234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-data\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428558 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.428291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-istio-token\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428558 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.428512 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-socket\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428661 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.428594 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-certs\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428661 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.428625 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-data\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428774 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.428744 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-credential-socket\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.428915 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.428890 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2578654c-9910-41d7-9015-c1bc5ac43911-istiod-ca-cert\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.430326 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.430303 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-envoy\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.430537 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.430523 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2578654c-9910-41d7-9015-c1bc5ac43911-istio-podinfo\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.439476 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.439456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njntm\" (UniqueName: \"kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-kube-api-access-njntm\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.439584 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.439482 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-istio-token\") pod \"e2e-mt-c488720f-openshift-default-79586c8695-65x68\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.489830 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.489805 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:20.634993 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.634967 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68"] Jul 14 22:24:20.635689 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:24:20.635660 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2578654c_9910_41d7_9015_c1bc5ac43911.slice/crio-7be37580308249f127bb56f3be894fe03eb116418f954a0bd16738d8d75f608f WatchSource:0}: Error finding container 7be37580308249f127bb56f3be894fe03eb116418f954a0bd16738d8d75f608f: Status 404 returned error can't find the container with id 7be37580308249f127bb56f3be894fe03eb116418f954a0bd16738d8d75f608f Jul 14 22:24:20.637816 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.637784 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:24:20.638222 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.637850 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:24:20.638222 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:20.637878 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:24:21.458208 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:21.458147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" event={"ID":"2578654c-9910-41d7-9015-c1bc5ac43911","Type":"ContainerStarted","Data":"7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7"} Jul 14 22:24:21.458599 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:21.458304 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" event={"ID":"2578654c-9910-41d7-9015-c1bc5ac43911","Type":"ContainerStarted","Data":"7be37580308249f127bb56f3be894fe03eb116418f954a0bd16738d8d75f608f"} Jul 14 22:24:21.485827 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:21.485774 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" podStartSLOduration=1.485757362 podStartE2EDuration="1.485757362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:24:21.483504786 +0000 UTC m=+1996.332974826" watchObservedRunningTime="2026-07-14 22:24:21.485757362 +0000 UTC m=+1996.335227401" Jul 14 22:24:21.490668 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:21.490642 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:21.495730 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:21.495709 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:22.463781 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:22.463750 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:22.464681 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:22.464662 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:35.618135 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.618101 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk"] Jul 14 22:24:35.618508 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.618471 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" Jul 14 22:24:35.618508 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.618482 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" Jul 14 22:24:35.618577 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.618546 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d4c5dc-950c-4de1-bd8d-2d8fcb7c2c39" containerName="istio-proxy" Jul 14 22:24:35.621693 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.621672 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" Jul 14 22:24:35.624378 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.624355 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:24:35.624509 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.624355 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:24:35.630141 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.629932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk"] Jul 14 22:24:35.769484 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.769445 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3132de05-86ab-4fa8-8089-7143a0b62b09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c488720f-5wmqk\" (UID: \"3132de05-86ab-4fa8-8089-7143a0b62b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" Jul 14 22:24:35.870967 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.870871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3132de05-86ab-4fa8-8089-7143a0b62b09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c488720f-5wmqk\" (UID: \"3132de05-86ab-4fa8-8089-7143a0b62b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" Jul 14 22:24:35.871535 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.871516 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3132de05-86ab-4fa8-8089-7143a0b62b09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c488720f-5wmqk\" (UID: \"3132de05-86ab-4fa8-8089-7143a0b62b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" Jul 14 22:24:35.935136 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:35.935095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" Jul 14 22:24:36.074674 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:36.074642 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk"] Jul 14 22:24:36.076228 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:24:36.076200 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3132de05_86ab_4fa8_8089_7143a0b62b09.slice/crio-b8c509fa34a973c4fd597aecd7e04bac3188867cab1f4d8da63f94b4bb3d2c88 WatchSource:0}: Error finding container b8c509fa34a973c4fd597aecd7e04bac3188867cab1f4d8da63f94b4bb3d2c88: Status 404 returned error can't find the container with id b8c509fa34a973c4fd597aecd7e04bac3188867cab1f4d8da63f94b4bb3d2c88 Jul 14 22:24:36.547191 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:36.547154 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" event={"ID":"3132de05-86ab-4fa8-8089-7143a0b62b09","Type":"ContainerStarted","Data":"60f205ac231e8cb21fdc78fc3d42eef081c93cb13759259bf118b65c4191bec1"} Jul 14 22:24:36.547359 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:36.547220 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" event={"ID":"3132de05-86ab-4fa8-8089-7143a0b62b09","Type":"ContainerStarted","Data":"b8c509fa34a973c4fd597aecd7e04bac3188867cab1f4d8da63f94b4bb3d2c88"} Jul 14 22:24:36.565484 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:36.565441 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" podStartSLOduration=1.565425265 podStartE2EDuration="1.565425265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:24:36.563283653 +0000 UTC m=+2011.412753693" watchObservedRunningTime="2026-07-14 22:24:36.565425265 +0000 UTC m=+2011.414895305" Jul 14 22:24:37.465333 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:37.465291 2562 patch_prober.go:28] interesting pod/e2e-mt-c488720f-openshift-default-79586c8695-65x68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 14 22:24:37.465761 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:37.465350 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 14 22:24:37.554477 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:37.554446 2562 generic.go:356] "Generic (PLEG): container finished" podID="2578654c-9910-41d7-9015-c1bc5ac43911" containerID="7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7" exitCode=0 Jul 14 22:24:37.554649 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:37.554519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" event={"ID":"2578654c-9910-41d7-9015-c1bc5ac43911","Type":"ContainerDied","Data":"7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7"} Jul 14 22:24:37.554994 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:37.554967 2562 scope.go:117] "RemoveContainer" containerID="7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7" Jul 14 22:24:37.556194 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:37.556155 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:24:37.556313 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:37.556251 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:24:37.556382 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:37.556312 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:24:37.556382 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:37.556365 2562 generic.go:356] "Generic (PLEG): container finished" podID="3132de05-86ab-4fa8-8089-7143a0b62b09" containerID="60f205ac231e8cb21fdc78fc3d42eef081c93cb13759259bf118b65c4191bec1" exitCode=0 Jul 14 22:24:37.556518 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:37.556413 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" event={"ID":"3132de05-86ab-4fa8-8089-7143a0b62b09","Type":"ContainerDied","Data":"60f205ac231e8cb21fdc78fc3d42eef081c93cb13759259bf118b65c4191bec1"} Jul 14 22:24:38.564460 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:38.564415 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" event={"ID":"2578654c-9910-41d7-9015-c1bc5ac43911","Type":"ContainerStarted","Data":"21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0"} Jul 14 22:24:38.734828 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:38.734805 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" Jul 14 22:24:38.907411 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:38.907314 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3132de05-86ab-4fa8-8089-7143a0b62b09-maas-api-service-ca\") pod \"3132de05-86ab-4fa8-8089-7143a0b62b09\" (UID: \"3132de05-86ab-4fa8-8089-7143a0b62b09\") " Jul 14 22:24:38.907681 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:38.907660 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3132de05-86ab-4fa8-8089-7143a0b62b09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3132de05-86ab-4fa8-8089-7143a0b62b09" (UID: "3132de05-86ab-4fa8-8089-7143a0b62b09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:24:39.008159 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:39.008121 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3132de05-86ab-4fa8-8089-7143a0b62b09-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:24:39.490609 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:39.490571 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:39.491011 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:39.490922 2562 patch_prober.go:28] interesting pod/e2e-mt-c488720f-openshift-default-79586c8695-65x68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 14 22:24:39.491233 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:39.491209 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 14 22:24:39.571275 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:39.571248 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" Jul 14 22:24:39.571674 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:39.571286 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c488720f-5wmqk" event={"ID":"3132de05-86ab-4fa8-8089-7143a0b62b09","Type":"ContainerDied","Data":"b8c509fa34a973c4fd597aecd7e04bac3188867cab1f4d8da63f94b4bb3d2c88"} Jul 14 22:24:39.571674 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:39.571333 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8c509fa34a973c4fd597aecd7e04bac3188867cab1f4d8da63f94b4bb3d2c88" Jul 14 22:24:40.490575 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:40.490532 2562 patch_prober.go:28] interesting pod/e2e-mt-c488720f-openshift-default-79586c8695-65x68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 14 22:24:40.490873 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:40.490843 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 14 22:24:41.491255 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:41.491215 2562 patch_prober.go:28] interesting pod/e2e-mt-c488720f-openshift-default-79586c8695-65x68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 14 22:24:41.491807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:41.491298 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 14 22:24:42.494807 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:42.494774 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:42.495237 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:42.495163 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:24:42.495817 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:24:42.495800 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:25:01.376304 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:01.376267 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68"] Jul 14 22:25:01.376841 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:01.376559 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" containerID="cri-o://21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0" gracePeriod=30 Jul 14 22:25:06.623767 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.623741 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:25:06.726514 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.726428 2562 generic.go:356] "Generic (PLEG): container finished" podID="2578654c-9910-41d7-9015-c1bc5ac43911" containerID="21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0" exitCode=0 Jul 14 22:25:06.726514 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.726475 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" Jul 14 22:25:06.726514 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.726487 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" event={"ID":"2578654c-9910-41d7-9015-c1bc5ac43911","Type":"ContainerDied","Data":"21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0"} Jul 14 22:25:06.726774 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.726529 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68" event={"ID":"2578654c-9910-41d7-9015-c1bc5ac43911","Type":"ContainerDied","Data":"7be37580308249f127bb56f3be894fe03eb116418f954a0bd16738d8d75f608f"} Jul 14 22:25:06.726774 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.726550 2562 scope.go:117] "RemoveContainer" containerID="21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0" Jul 14 22:25:06.735947 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.735930 2562 scope.go:117] "RemoveContainer" containerID="7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7" Jul 14 22:25:06.747382 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.747363 2562 scope.go:117] "RemoveContainer" containerID="21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0" Jul 14 22:25:06.747641 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:25:06.747618 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0\": container with ID starting with 21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0 not found: ID does not exist" containerID="21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0" Jul 14 22:25:06.747711 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.747651 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0"} err="failed to get container status \"21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0\": rpc error: code = NotFound desc = could not find container \"21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0\": container with ID starting with 21bb343068d3ef10ee73cd937f47511c9341a2be1da4ff7de133932ec3e00bf0 not found: ID does not exist" Jul 14 22:25:06.747711 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.747670 2562 scope.go:117] "RemoveContainer" containerID="7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7" Jul 14 22:25:06.747895 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:25:06.747878 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7\": container with ID starting with 7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7 not found: ID does not exist" containerID="7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7" Jul 14 22:25:06.747931 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.747899 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7"} err="failed to get container status \"7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7\": rpc error: code = NotFound desc = could not find container \"7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7\": container with ID starting with 7b9478ceeddcba06e09bd7050e45f85b2f1c458bd42cad78586eb6f1db5636c7 not found: ID does not exist" Jul 14 22:25:06.757222 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757200 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2578654c-9910-41d7-9015-c1bc5ac43911-istio-podinfo\") pod \"2578654c-9910-41d7-9015-c1bc5ac43911\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " Jul 14 22:25:06.757297 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757233 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2578654c-9910-41d7-9015-c1bc5ac43911-istiod-ca-cert\") pod \"2578654c-9910-41d7-9015-c1bc5ac43911\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " Jul 14 22:25:06.757297 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757253 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-socket\") pod \"2578654c-9910-41d7-9015-c1bc5ac43911\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " Jul 14 22:25:06.757374 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757300 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-credential-socket\") pod \"2578654c-9910-41d7-9015-c1bc5ac43911\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " Jul 14 22:25:06.757374 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757324 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-envoy\") pod \"2578654c-9910-41d7-9015-c1bc5ac43911\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " Jul 14 22:25:06.757374 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757344 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-certs\") pod \"2578654c-9910-41d7-9015-c1bc5ac43911\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " Jul 14 22:25:06.757374 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757373 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-istio-token\") pod \"2578654c-9910-41d7-9015-c1bc5ac43911\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " Jul 14 22:25:06.757571 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757397 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-data\") pod \"2578654c-9910-41d7-9015-c1bc5ac43911\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " Jul 14 22:25:06.757571 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757424 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njntm\" (UniqueName: \"kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-kube-api-access-njntm\") pod \"2578654c-9910-41d7-9015-c1bc5ac43911\" (UID: \"2578654c-9910-41d7-9015-c1bc5ac43911\") " Jul 14 22:25:06.757571 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757536 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2578654c-9910-41d7-9015-c1bc5ac43911" (UID: "2578654c-9910-41d7-9015-c1bc5ac43911"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:25:06.757742 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757610 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2578654c-9910-41d7-9015-c1bc5ac43911" (UID: "2578654c-9910-41d7-9015-c1bc5ac43911"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:25:06.757742 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757619 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2578654c-9910-41d7-9015-c1bc5ac43911" (UID: "2578654c-9910-41d7-9015-c1bc5ac43911"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:25:06.757903 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757787 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:25:06.757903 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757809 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:25:06.757903 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757824 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:25:06.758085 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.757918 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2578654c-9910-41d7-9015-c1bc5ac43911-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2578654c-9910-41d7-9015-c1bc5ac43911" (UID: "2578654c-9910-41d7-9015-c1bc5ac43911"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:25:06.758401 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.758378 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-data" (OuterVolumeSpecName: "istio-data") pod "2578654c-9910-41d7-9015-c1bc5ac43911" (UID: "2578654c-9910-41d7-9015-c1bc5ac43911"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:25:06.759501 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.759476 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-kube-api-access-njntm" (OuterVolumeSpecName: "kube-api-access-njntm") pod "2578654c-9910-41d7-9015-c1bc5ac43911" (UID: "2578654c-9910-41d7-9015-c1bc5ac43911"). InnerVolumeSpecName "kube-api-access-njntm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:25:06.759501 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.759493 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2578654c-9910-41d7-9015-c1bc5ac43911" (UID: "2578654c-9910-41d7-9015-c1bc5ac43911"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:25:06.759685 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.759669 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2578654c-9910-41d7-9015-c1bc5ac43911-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2578654c-9910-41d7-9015-c1bc5ac43911" (UID: "2578654c-9910-41d7-9015-c1bc5ac43911"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:25:06.759836 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.759818 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-istio-token" (OuterVolumeSpecName: "istio-token") pod "2578654c-9910-41d7-9015-c1bc5ac43911" (UID: "2578654c-9910-41d7-9015-c1bc5ac43911"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:25:06.858933 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.858904 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:25:06.858933 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.858930 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:25:06.858933 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.858938 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2578654c-9910-41d7-9015-c1bc5ac43911-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:25:06.859162 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.858948 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njntm\" (UniqueName: \"kubernetes.io/projected/2578654c-9910-41d7-9015-c1bc5ac43911-kube-api-access-njntm\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:25:06.859162 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.858962 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2578654c-9910-41d7-9015-c1bc5ac43911-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:25:06.859162 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.858971 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2578654c-9910-41d7-9015-c1bc5ac43911-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:25:06.976251 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:06.976224 2562 scope.go:117] "RemoveContainer" containerID="5816a226e390c7cd0ab537d1b490bc1797beea2fc306051eacd6328261749ca0" Jul 14 22:25:07.045004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:07.044975 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68"] Jul 14 22:25:07.053869 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:07.051579 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c488720f-openshift-default-79586c8695-65x68"] Jul 14 22:25:08.810608 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:08.810576 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" path="/var/lib/kubelet/pods/2578654c-9910-41d7-9015-c1bc5ac43911/volumes" Jul 14 22:25:20.982447 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.982415 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88"] Jul 14 22:25:20.982950 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.982814 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" Jul 14 22:25:20.982950 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.982828 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" Jul 14 22:25:20.982950 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.982847 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" Jul 14 22:25:20.982950 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.982853 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" Jul 14 22:25:20.982950 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.982860 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3132de05-86ab-4fa8-8089-7143a0b62b09" containerName="revoke-keys" Jul 14 22:25:20.982950 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.982867 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3132de05-86ab-4fa8-8089-7143a0b62b09" containerName="revoke-keys" Jul 14 22:25:20.982950 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.982931 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" Jul 14 22:25:20.982950 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.982937 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2578654c-9910-41d7-9015-c1bc5ac43911" containerName="istio-proxy" Jul 14 22:25:20.982950 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.982945 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3132de05-86ab-4fa8-8089-7143a0b62b09" containerName="revoke-keys" Jul 14 22:25:20.986126 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.986107 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:20.989391 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.989368 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ba078dbe-openshift-default-dockercfg-w8vl6\"" Jul 14 22:25:20.997605 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:20.997572 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88"] Jul 14 22:25:21.087575 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.087539 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-data\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.087770 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.087584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istiod-ca-cert\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.087770 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.087633 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-envoy\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.087770 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.087701 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-socket\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.087770 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.087725 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-certs\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.087770 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.087741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-podinfo\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.087770 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.087764 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5jd\" (UniqueName: \"kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-kube-api-access-pz5jd\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.088130 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.087876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-credential-socket\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.088130 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.087926 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-token\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.188838 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.188801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-token\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.188852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-data\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.188870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istiod-ca-cert\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.188905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-envoy\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.188929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-socket\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.188945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-certs\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.188960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-podinfo\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.188975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5jd\" (UniqueName: \"kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-kube-api-access-pz5jd\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.189050 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-credential-socket\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189403 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.189256 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-data\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189462 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.189443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-credential-socket\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189530 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.189493 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-certs\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189582 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.189526 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-socket\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.189864 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.189847 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istiod-ca-cert\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.191491 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.191469 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-envoy\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.191637 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.191618 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-podinfo\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.197783 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.197764 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-token\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.197922 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.197809 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5jd\" (UniqueName: \"kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-kube-api-access-pz5jd\") pod \"e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.296925 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.296893 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:21.437782 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.437750 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88"] Jul 14 22:25:21.439098 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:25:21.439073 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdc3c65_25e9_4115_9bf6_0ed58b88ca72.slice/crio-88c1668efa3efd9193fc7a30eca3a446973e0a7604849ef43c71b9a795bb2922 WatchSource:0}: Error finding container 88c1668efa3efd9193fc7a30eca3a446973e0a7604849ef43c71b9a795bb2922: Status 404 returned error can't find the container with id 88c1668efa3efd9193fc7a30eca3a446973e0a7604849ef43c71b9a795bb2922 Jul 14 22:25:21.441429 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.441382 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:25:21.441515 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.441488 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:25:21.441555 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.441534 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:25:21.811468 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.811430 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" event={"ID":"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72","Type":"ContainerStarted","Data":"7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c"} Jul 14 22:25:21.811632 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:21.811496 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" event={"ID":"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72","Type":"ContainerStarted","Data":"88c1668efa3efd9193fc7a30eca3a446973e0a7604849ef43c71b9a795bb2922"} Jul 14 22:25:22.297523 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:22.297489 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:22.302323 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:22.302298 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:22.323358 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:22.323306 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" podStartSLOduration=2.323289353 podStartE2EDuration="2.323289353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:25:21.832092258 +0000 UTC m=+2056.681562300" watchObservedRunningTime="2026-07-14 22:25:22.323289353 +0000 UTC m=+2057.172759463" Jul 14 22:25:22.817980 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:22.817949 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:22.818988 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:22.818968 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:37.819870 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:37.819835 2562 patch_prober.go:28] interesting pod/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 14 22:25:37.820325 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:37.819907 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 14 22:25:37.909465 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:37.909430 2562 generic.go:356] "Generic (PLEG): container finished" podID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerID="7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c" exitCode=0 Jul 14 22:25:37.909630 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:37.909500 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" event={"ID":"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72","Type":"ContainerDied","Data":"7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c"} Jul 14 22:25:37.909901 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:37.909888 2562 scope.go:117] "RemoveContainer" containerID="7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c" Jul 14 22:25:37.910938 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:37.910910 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:25:37.911067 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:37.910992 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:25:37.911140 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:37.911074 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:25:38.917357 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:38.917323 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" event={"ID":"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72","Type":"ContainerStarted","Data":"67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824"} Jul 14 22:25:39.297368 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:39.297323 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:39.297595 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:39.297466 2562 patch_prober.go:28] interesting pod/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 14 22:25:39.297595 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:39.297512 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 14 22:25:40.297983 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:40.297941 2562 patch_prober.go:28] interesting pod/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 14 22:25:40.298628 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:40.298595 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 14 22:25:41.298179 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:41.298132 2562 patch_prober.go:28] interesting pod/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 14 22:25:41.298825 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:41.298781 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 14 22:25:42.297563 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:42.297513 2562 patch_prober.go:28] interesting pod/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 14 22:25:42.297787 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:42.297598 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 14 22:25:43.302182 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:43.302149 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:43.302625 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:43.302474 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:25:43.303314 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:25:43.303296 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:26:06.731405 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:06.731373 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc"] Jul 14 22:26:06.734984 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:06.734961 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc" Jul 14 22:26:06.742495 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:06.742328 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:26:06.742903 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:06.742512 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:26:06.746959 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:06.746934 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc"] Jul 14 22:26:06.798802 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:06.798769 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a43c3e-1703-48a2-9bb5-7cd8896e03da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba078dbe-72snc\" (UID: \"53a43c3e-1703-48a2-9bb5-7cd8896e03da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc" Jul 14 22:26:06.899295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:06.899269 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a43c3e-1703-48a2-9bb5-7cd8896e03da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba078dbe-72snc\" (UID: \"53a43c3e-1703-48a2-9bb5-7cd8896e03da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc" Jul 14 22:26:06.899948 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:06.899920 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a43c3e-1703-48a2-9bb5-7cd8896e03da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba078dbe-72snc\" (UID: \"53a43c3e-1703-48a2-9bb5-7cd8896e03da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc" Jul 14 22:26:07.051278 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:07.051247 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc" Jul 14 22:26:07.193167 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:07.193143 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc"] Jul 14 22:26:07.194771 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:26:07.194745 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a43c3e_1703_48a2_9bb5_7cd8896e03da.slice/crio-8adbadab931d5f2a8e634e0ae58a76a082eda9af753cb5505ce92940a18f179e WatchSource:0}: Error finding container 8adbadab931d5f2a8e634e0ae58a76a082eda9af753cb5505ce92940a18f179e: Status 404 returned error can't find the container with id 8adbadab931d5f2a8e634e0ae58a76a082eda9af753cb5505ce92940a18f179e Jul 14 22:26:08.081958 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:08.081916 2562 generic.go:356] "Generic (PLEG): container finished" podID="53a43c3e-1703-48a2-9bb5-7cd8896e03da" containerID="a2b3fb3474efb7e66022dc66f375dbfb63f8174fa4389715e29aaf3c59bb4831" exitCode=0 Jul 14 22:26:08.082422 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:08.081989 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc" event={"ID":"53a43c3e-1703-48a2-9bb5-7cd8896e03da","Type":"ContainerDied","Data":"a2b3fb3474efb7e66022dc66f375dbfb63f8174fa4389715e29aaf3c59bb4831"} Jul 14 22:26:08.082422 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:08.082052 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc" event={"ID":"53a43c3e-1703-48a2-9bb5-7cd8896e03da","Type":"ContainerStarted","Data":"8adbadab931d5f2a8e634e0ae58a76a082eda9af753cb5505ce92940a18f179e"} Jul 14 22:26:09.223947 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:09.223918 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc" Jul 14 22:26:09.320378 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:09.320343 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a43c3e-1703-48a2-9bb5-7cd8896e03da-maas-api-service-ca\") pod \"53a43c3e-1703-48a2-9bb5-7cd8896e03da\" (UID: \"53a43c3e-1703-48a2-9bb5-7cd8896e03da\") " Jul 14 22:26:09.320726 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:09.320702 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a43c3e-1703-48a2-9bb5-7cd8896e03da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53a43c3e-1703-48a2-9bb5-7cd8896e03da" (UID: "53a43c3e-1703-48a2-9bb5-7cd8896e03da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:26:09.421146 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:09.421065 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a43c3e-1703-48a2-9bb5-7cd8896e03da-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:26:10.094422 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:10.094385 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc" event={"ID":"53a43c3e-1703-48a2-9bb5-7cd8896e03da","Type":"ContainerDied","Data":"8adbadab931d5f2a8e634e0ae58a76a082eda9af753cb5505ce92940a18f179e"} Jul 14 22:26:10.094422 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:10.094420 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba078dbe-72snc" Jul 14 22:26:10.094639 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:26:10.094441 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8adbadab931d5f2a8e634e0ae58a76a082eda9af753cb5505ce92940a18f179e" Jul 14 22:27:07.010345 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:07.010254 2562 scope.go:117] "RemoveContainer" containerID="3450c6c8a3b9eeb27409a38c75d51e5d0371cb3711e38e0a5e38ae4cfa18c049" Jul 14 22:27:14.093882 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:14.093845 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88"] Jul 14 22:27:14.094876 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:14.094838 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" containerID="cri-o://67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824" gracePeriod=30 Jul 14 22:27:19.345977 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.345954 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:27:19.438719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.438632 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-credential-socket\") pod \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " Jul 14 22:27:19.438719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.438676 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-token\") pod \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " Jul 14 22:27:19.438961 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.438747 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-data\") pod \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " Jul 14 22:27:19.438961 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.438773 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-envoy\") pod \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " Jul 14 22:27:19.438961 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.438795 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-certs\") pod \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " Jul 14 22:27:19.438961 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.438827 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-podinfo\") pod \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " Jul 14 22:27:19.438961 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.438881 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istiod-ca-cert\") pod \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " Jul 14 22:27:19.438961 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.438935 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-socket\") pod \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " Jul 14 22:27:19.439309 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.438982 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" (UID: "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:27:19.439309 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.438989 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz5jd\" (UniqueName: \"kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-kube-api-access-pz5jd\") pod \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\" (UID: \"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72\") " Jul 14 22:27:19.439309 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.439231 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" (UID: "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:27:19.439463 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.439329 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:27:19.439463 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.439347 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:27:19.439463 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.439435 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" (UID: "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:27:19.439621 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.439533 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" (UID: "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:27:19.439677 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.439616 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" (UID: "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:27:19.441188 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.441161 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" (UID: "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:27:19.441293 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.441189 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" (UID: "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:27:19.441293 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.441257 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-kube-api-access-pz5jd" (OuterVolumeSpecName: "kube-api-access-pz5jd") pod "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" (UID: "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72"). InnerVolumeSpecName "kube-api-access-pz5jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:27:19.441398 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.441314 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" (UID: "dbdc3c65-25e9-4115-9bf6-0ed58b88ca72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:27:19.486169 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.486131 2562 generic.go:356] "Generic (PLEG): container finished" podID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerID="67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824" exitCode=0 Jul 14 22:27:19.486303 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.486190 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" Jul 14 22:27:19.486303 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.486213 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" event={"ID":"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72","Type":"ContainerDied","Data":"67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824"} Jul 14 22:27:19.486303 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.486265 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88" event={"ID":"dbdc3c65-25e9-4115-9bf6-0ed58b88ca72","Type":"ContainerDied","Data":"88c1668efa3efd9193fc7a30eca3a446973e0a7604849ef43c71b9a795bb2922"} Jul 14 22:27:19.486303 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.486297 2562 scope.go:117] "RemoveContainer" containerID="67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824" Jul 14 22:27:19.495439 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.495412 2562 scope.go:117] "RemoveContainer" containerID="7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c" Jul 14 22:27:19.503810 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.503794 2562 scope.go:117] "RemoveContainer" containerID="67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824" Jul 14 22:27:19.504063 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:27:19.504018 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824\": container with ID starting with 67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824 not found: ID does not exist" containerID="67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824" Jul 14 22:27:19.504167 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.504070 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824"} err="failed to get container status \"67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824\": rpc error: code = NotFound desc = could not find container \"67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824\": container with ID starting with 67208f539e116729bdca11e6ed1628e66c6c95ae1f6ace82115800ad3ea5e824 not found: ID does not exist" Jul 14 22:27:19.504167 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.504089 2562 scope.go:117] "RemoveContainer" containerID="7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c" Jul 14 22:27:19.504338 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:27:19.504323 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c\": container with ID starting with 7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c not found: ID does not exist" containerID="7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c" Jul 14 22:27:19.504382 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.504343 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c"} err="failed to get container status \"7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c\": rpc error: code = NotFound desc = could not find container \"7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c\": container with ID starting with 7fe107766625e5df45fa04872a30f66c2c770add735f72ec0eebfc7bdc67802c not found: ID does not exist" Jul 14 22:27:19.509696 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.509674 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88"] Jul 14 22:27:19.516310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.516286 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ba078dbe-openshift-default-7c66f555f-6rx88"] Jul 14 22:27:19.540423 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.540398 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:27:19.540423 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.540422 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:27:19.540572 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.540436 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:27:19.540572 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.540448 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:27:19.540572 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.540460 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:27:19.540572 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.540474 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz5jd\" (UniqueName: \"kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-kube-api-access-pz5jd\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:27:19.540572 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:19.540487 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:27:20.812277 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:27:20.812244 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" path="/var/lib/kubelet/pods/dbdc3c65-25e9-4115-9bf6-0ed58b88ca72/volumes" Jul 14 22:28:07.032087 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:07.032055 2562 scope.go:117] "RemoveContainer" containerID="29104b53e718932c2a48237cb6d20b08cf5194a2ae6f79dfd8063fa35e3377fd" Jul 14 22:28:09.973201 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.973169 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h"] Jul 14 22:28:09.973582 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.973545 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53a43c3e-1703-48a2-9bb5-7cd8896e03da" containerName="revoke-keys" Jul 14 22:28:09.973582 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.973560 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53a43c3e-1703-48a2-9bb5-7cd8896e03da" containerName="revoke-keys" Jul 14 22:28:09.973582 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.973576 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" Jul 14 22:28:09.973582 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.973581 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" Jul 14 22:28:09.973719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.973593 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" Jul 14 22:28:09.973719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.973600 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" Jul 14 22:28:09.973719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.973677 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="53a43c3e-1703-48a2-9bb5-7cd8896e03da" containerName="revoke-keys" Jul 14 22:28:09.973719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.973686 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" Jul 14 22:28:09.973719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.973692 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdc3c65-25e9-4115-9bf6-0ed58b88ca72" containerName="istio-proxy" Jul 14 22:28:09.978103 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.978084 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:09.981821 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.981803 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6a0cfb-openshift-default-dockercfg-72zgm\"" Jul 14 22:28:09.991828 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:09.991800 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h"] Jul 14 22:28:10.076826 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.076793 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-certs\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.076826 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.076825 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-envoy\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.077129 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.076853 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-socket\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.077129 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.076944 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-credential-socket\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.077129 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.076984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/104430b4-2dbc-48f0-b96e-867018353f63-istio-podinfo\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.077129 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.077068 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-istio-token\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.077129 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.077121 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-data\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.077347 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.077159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/104430b4-2dbc-48f0-b96e-867018353f63-istiod-ca-cert\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.077347 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.077186 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2n4s\" (UniqueName: \"kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-kube-api-access-d2n4s\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.177739 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.177697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-certs\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.177739 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.177738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-envoy\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.177961 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.177759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-socket\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.177961 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.177894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-credential-socket\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.177961 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.177935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/104430b4-2dbc-48f0-b96e-867018353f63-istio-podinfo\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.178095 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.177997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-istio-token\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.178095 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.178084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-data\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.178199 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.178130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/104430b4-2dbc-48f0-b96e-867018353f63-istiod-ca-cert\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.178199 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.178158 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2n4s\" (UniqueName: \"kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-kube-api-access-d2n4s\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.178199 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.178168 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-socket\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.178342 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.178209 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-certs\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.178342 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.178295 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-credential-socket\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.178493 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.178474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-data\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.178892 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.178871 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/104430b4-2dbc-48f0-b96e-867018353f63-istiod-ca-cert\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.180350 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.180318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-envoy\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.180544 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.180520 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/104430b4-2dbc-48f0-b96e-867018353f63-istio-podinfo\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.189749 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.189730 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-istio-token\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.190358 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.190340 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2n4s\" (UniqueName: \"kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-kube-api-access-d2n4s\") pod \"e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.287904 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.287868 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:10.436108 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.436074 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h"] Jul 14 22:28:10.437058 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:28:10.437010 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104430b4_2dbc_48f0_b96e_867018353f63.slice/crio-c9de4faf2cdbcd99e2e27cd8470f28874570186713a787b3db48d693f73f5205 WatchSource:0}: Error finding container c9de4faf2cdbcd99e2e27cd8470f28874570186713a787b3db48d693f73f5205: Status 404 returned error can't find the container with id c9de4faf2cdbcd99e2e27cd8470f28874570186713a787b3db48d693f73f5205 Jul 14 22:28:10.439104 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.439067 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:10.439210 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.439155 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:10.439210 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.439200 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:10.776181 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.776146 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" event={"ID":"104430b4-2dbc-48f0-b96e-867018353f63","Type":"ContainerStarted","Data":"16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018"} Jul 14 22:28:10.776352 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.776199 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" event={"ID":"104430b4-2dbc-48f0-b96e-867018353f63","Type":"ContainerStarted","Data":"c9de4faf2cdbcd99e2e27cd8470f28874570186713a787b3db48d693f73f5205"} Jul 14 22:28:10.822553 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:10.822496 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" podStartSLOduration=1.822480669 podStartE2EDuration="1.822480669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:28:10.818847899 +0000 UTC m=+2225.668317939" watchObservedRunningTime="2026-07-14 22:28:10.822480669 +0000 UTC m=+2225.671950708" Jul 14 22:28:11.288489 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:11.288449 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:11.293628 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:11.293601 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:11.782082 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:11.782049 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:11.783061 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:11.783041 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:18.015408 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.015368 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x"] Jul 14 22:28:18.019064 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.019040 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.022714 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.022694 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b4e43e-openshift-default-dockercfg-2ksw4\"" Jul 14 22:28:18.035240 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.035215 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x"] Jul 14 22:28:18.149452 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.149409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istiod-ca-cert\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.149452 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.149450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-certs\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.149688 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.149479 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-podinfo\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.149688 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.149572 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflkk\" (UniqueName: \"kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-kube-api-access-rflkk\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.149688 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.149628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-credential-socket\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.149688 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.149661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-data\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.149836 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.149693 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-socket\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.149836 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.149709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-token\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.149836 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.149751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-envoy\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.250473 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istiod-ca-cert\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.250473 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-certs\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.250691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-podinfo\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.250691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflkk\" (UniqueName: \"kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-kube-api-access-rflkk\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.250691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250597 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-credential-socket\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.250691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-data\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.250691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250673 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-socket\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.250940 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-token\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.250940 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-envoy\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.250940 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250888 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-certs\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.251146 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.250984 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-credential-socket\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.251244 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.251211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-socket\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.251372 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.251269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istiod-ca-cert\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.251455 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.251432 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-data\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.252931 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.252912 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-envoy\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.253186 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.253170 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-podinfo\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.260672 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.260647 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflkk\" (UniqueName: \"kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-kube-api-access-rflkk\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.262590 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.262572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-token\") pod \"e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.330885 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.330845 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:18.494051 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.494008 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x"] Jul 14 22:28:18.495372 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:28:18.495343 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0794d30e_7f50_4c8e_84ab_06fbe68afe04.slice/crio-f60c76aecd153c9ed0c48f954154e6d438a827387cd46c375bd926fbb8b69ae4 WatchSource:0}: Error finding container f60c76aecd153c9ed0c48f954154e6d438a827387cd46c375bd926fbb8b69ae4: Status 404 returned error can't find the container with id f60c76aecd153c9ed0c48f954154e6d438a827387cd46c375bd926fbb8b69ae4 Jul 14 22:28:18.497507 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.497473 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:18.497580 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.497552 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:18.497622 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.497589 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:18.829645 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.829592 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" event={"ID":"0794d30e-7f50-4c8e-84ab-06fbe68afe04","Type":"ContainerStarted","Data":"4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa"} Jul 14 22:28:18.829883 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.829675 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" event={"ID":"0794d30e-7f50-4c8e-84ab-06fbe68afe04","Type":"ContainerStarted","Data":"f60c76aecd153c9ed0c48f954154e6d438a827387cd46c375bd926fbb8b69ae4"} Jul 14 22:28:18.858965 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:18.858915 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" podStartSLOduration=1.858900331 podStartE2EDuration="1.858900331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:28:18.856502534 +0000 UTC m=+2233.705972574" watchObservedRunningTime="2026-07-14 22:28:18.858900331 +0000 UTC m=+2233.708370370" Jul 14 22:28:19.331830 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:19.331795 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:19.336814 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:19.336790 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:19.835479 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:19.835449 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:19.836703 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:19.836678 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:27.824231 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:27.824132 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 22:28:27.824231 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:27.824199 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 22:28:29.897671 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:29.897627 2562 generic.go:356] "Generic (PLEG): container finished" podID="104430b4-2dbc-48f0-b96e-867018353f63" containerID="16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018" exitCode=0 Jul 14 22:28:29.898250 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:29.897792 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" event={"ID":"104430b4-2dbc-48f0-b96e-867018353f63","Type":"ContainerDied","Data":"16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018"} Jul 14 22:28:29.898360 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:29.898339 2562 scope.go:117] "RemoveContainer" containerID="16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018" Jul 14 22:28:29.899934 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:29.899890 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:29.900066 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:29.899985 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:29.900066 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:29.900050 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:30.909510 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:30.909462 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" event={"ID":"104430b4-2dbc-48f0-b96e-867018353f63","Type":"ContainerStarted","Data":"00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d"} Jul 14 22:28:31.288520 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:31.288478 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:31.288855 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:31.288817 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 14 22:28:31.288960 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:31.288889 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 14 22:28:32.289384 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:32.289341 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 14 22:28:32.290075 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:32.290039 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 14 22:28:33.289727 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:33.289280 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 14 22:28:33.289727 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:33.289339 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 14 22:28:34.289134 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:34.289090 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 14 22:28:34.289427 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:34.289397 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 14 22:28:35.292112 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:35.292082 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:35.292501 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:35.292372 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:35.293098 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:35.293082 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:28:35.877255 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:35.877214 2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 22:28:35.878266 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:35.877307 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 22:28:36.951418 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:36.951383 2562 generic.go:356] "Generic (PLEG): container finished" podID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerID="4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa" exitCode=0 Jul 14 22:28:36.951820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:36.951458 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" event={"ID":"0794d30e-7f50-4c8e-84ab-06fbe68afe04","Type":"ContainerDied","Data":"4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa"} Jul 14 22:28:36.951952 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:36.951938 2562 scope.go:117] "RemoveContainer" containerID="4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa" Jul 14 22:28:36.952930 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:36.952869 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:36.953009 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:36.952935 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:36.953009 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:36.952969 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:28:37.958564 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:37.958529 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" event={"ID":"0794d30e-7f50-4c8e-84ab-06fbe68afe04","Type":"ContainerStarted","Data":"2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e"} Jul 14 22:28:38.331775 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:38.331730 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:38.331998 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:38.331978 2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 14 22:28:38.332108 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:38.332035 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 14 22:28:39.331895 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:39.331856 2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 14 22:28:39.332359 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:39.331914 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 14 22:28:40.331607 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:40.331564 2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 14 22:28:40.331808 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:40.331624 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 14 22:28:41.331704 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:41.331669 2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 14 22:28:41.332146 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:41.331725 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 14 22:28:42.335460 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:42.335428 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:42.335853 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:42.335782 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:28:42.336415 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:28:42.336397 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:29:35.763015 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:35.762981 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv"] Jul 14 22:29:35.766470 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:35.766452 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv" Jul 14 22:29:35.769162 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:35.769132 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:29:35.769330 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:35.769274 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:29:35.778045 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:35.776817 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv"] Jul 14 22:29:35.876999 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:35.876961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv\" (UID: \"c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv" Jul 14 22:29:35.978402 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:35.978358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv\" (UID: \"c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv" Jul 14 22:29:35.979042 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:35.978999 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv\" (UID: \"c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv" Jul 14 22:29:36.086432 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:36.086392 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv" Jul 14 22:29:36.221633 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:36.221602 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv"] Jul 14 22:29:36.222881 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:29:36.222858 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c5b0e2_b877_4ddc_8174_0e3a9180c7cf.slice/crio-0f4b4dec4112be1efd2c7f0c4c8c32371edda54af2fbc9b031cb8731cc880d7c WatchSource:0}: Error finding container 0f4b4dec4112be1efd2c7f0c4c8c32371edda54af2fbc9b031cb8731cc880d7c: Status 404 returned error can't find the container with id 0f4b4dec4112be1efd2c7f0c4c8c32371edda54af2fbc9b031cb8731cc880d7c Jul 14 22:29:36.293743 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:36.293711 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv" event={"ID":"c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf","Type":"ContainerStarted","Data":"0f4b4dec4112be1efd2c7f0c4c8c32371edda54af2fbc9b031cb8731cc880d7c"} Jul 14 22:29:37.301273 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:37.301232 2562 generic.go:356] "Generic (PLEG): container finished" podID="c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf" containerID="8b5b169b34f8971b88d982edd9f610e9bdb49247a3fc2046c431fc683cd263e9" exitCode=0 Jul 14 22:29:37.301668 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:37.301274 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv" event={"ID":"c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf","Type":"ContainerDied","Data":"8b5b169b34f8971b88d982edd9f610e9bdb49247a3fc2046c431fc683cd263e9"} Jul 14 22:29:38.435875 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:38.435854 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv" Jul 14 22:29:38.601783 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:38.601688 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf-maas-api-service-ca\") pod \"c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf\" (UID: \"c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf\") " Jul 14 22:29:38.602134 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:38.602108 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf" (UID: "c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:29:38.702848 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:38.702814 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:29:39.314074 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:39.314014 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv" Jul 14 22:29:39.314074 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:39.314065 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv" event={"ID":"c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf","Type":"ContainerDied","Data":"0f4b4dec4112be1efd2c7f0c4c8c32371edda54af2fbc9b031cb8731cc880d7c"} Jul 14 22:29:39.314307 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:29:39.314119 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f4b4dec4112be1efd2c7f0c4c8c32371edda54af2fbc9b031cb8731cc880d7c" Jul 14 22:30:00.146470 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:00.146388 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9"] Jul 14 22:30:00.146844 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:00.146794 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf" containerName="revoke-keys" Jul 14 22:30:00.146844 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:00.146805 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf" containerName="revoke-keys" Jul 14 22:30:00.146921 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:00.146863 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf" containerName="revoke-keys" Jul 14 22:30:00.150074 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:00.150051 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9" Jul 14 22:30:00.153140 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:00.153117 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:30:00.159961 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:00.159935 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9"] Jul 14 22:30:00.160801 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:00.160782 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9" Jul 14 22:30:00.301417 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:00.301391 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9"] Jul 14 22:30:00.302829 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:30:00.302802 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e7772c_387c_4f52_9967_0eff6af6e49a.slice/crio-7fe7aeef9698c7a1f8092c6ea17c532e49b7bfceab3c9846b59ba9e8b2f9e313 WatchSource:0}: Error finding container 7fe7aeef9698c7a1f8092c6ea17c532e49b7bfceab3c9846b59ba9e8b2f9e313: Status 404 returned error can't find the container with id 7fe7aeef9698c7a1f8092c6ea17c532e49b7bfceab3c9846b59ba9e8b2f9e313 Jul 14 22:30:00.439630 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:00.439534 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9" event={"ID":"30e7772c-387c-4f52-9967-0eff6af6e49a","Type":"ContainerStarted","Data":"7fe7aeef9698c7a1f8092c6ea17c532e49b7bfceab3c9846b59ba9e8b2f9e313"} Jul 14 22:30:01.447074 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:01.446972 2562 generic.go:356] "Generic (PLEG): container finished" podID="30e7772c-387c-4f52-9967-0eff6af6e49a" containerID="47a8649ad0c5c8f79dbffb542013fba335b024af0f87978d18528bdee68a9821" exitCode=0 Jul 14 22:30:01.447462 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:01.447068 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9" event={"ID":"30e7772c-387c-4f52-9967-0eff6af6e49a","Type":"ContainerDied","Data":"47a8649ad0c5c8f79dbffb542013fba335b024af0f87978d18528bdee68a9821"} Jul 14 22:30:01.575485 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:01.575449 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h"] Jul 14 22:30:01.575686 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:01.575665 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" containerID="cri-o://00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d" gracePeriod=30 Jul 14 22:30:02.586781 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:02.586757 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9" Jul 14 22:30:03.459189 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:03.459158 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9" Jul 14 22:30:03.459376 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:03.459192 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9" event={"ID":"30e7772c-387c-4f52-9967-0eff6af6e49a","Type":"ContainerDied","Data":"7fe7aeef9698c7a1f8092c6ea17c532e49b7bfceab3c9846b59ba9e8b2f9e313"} Jul 14 22:30:03.459376 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:03.459241 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fe7aeef9698c7a1f8092c6ea17c532e49b7bfceab3c9846b59ba9e8b2f9e313" Jul 14 22:30:05.295019 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:05.294981 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:30:05.295446 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:05.295063 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:30:06.837078 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.837004 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:30:06.955159 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955131 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-socket\") pod \"104430b4-2dbc-48f0-b96e-867018353f63\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " Jul 14 22:30:06.955159 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955162 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/104430b4-2dbc-48f0-b96e-867018353f63-istio-podinfo\") pod \"104430b4-2dbc-48f0-b96e-867018353f63\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " Jul 14 22:30:06.955406 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955190 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-certs\") pod \"104430b4-2dbc-48f0-b96e-867018353f63\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " Jul 14 22:30:06.955406 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955211 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-envoy\") pod \"104430b4-2dbc-48f0-b96e-867018353f63\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " Jul 14 22:30:06.955406 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955250 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/104430b4-2dbc-48f0-b96e-867018353f63-istiod-ca-cert\") pod \"104430b4-2dbc-48f0-b96e-867018353f63\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " Jul 14 22:30:06.955406 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955324 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-data\") pod \"104430b4-2dbc-48f0-b96e-867018353f63\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " Jul 14 22:30:06.955406 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955370 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-istio-token\") pod \"104430b4-2dbc-48f0-b96e-867018353f63\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " Jul 14 22:30:06.955406 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955395 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-credential-socket\") pod \"104430b4-2dbc-48f0-b96e-867018353f63\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " Jul 14 22:30:06.955703 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955456 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2n4s\" (UniqueName: \"kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-kube-api-access-d2n4s\") pod \"104430b4-2dbc-48f0-b96e-867018353f63\" (UID: \"104430b4-2dbc-48f0-b96e-867018353f63\") " Jul 14 22:30:06.955703 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955458 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "104430b4-2dbc-48f0-b96e-867018353f63" (UID: "104430b4-2dbc-48f0-b96e-867018353f63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:30:06.955703 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955560 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "104430b4-2dbc-48f0-b96e-867018353f63" (UID: "104430b4-2dbc-48f0-b96e-867018353f63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:30:06.955854 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955772 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:06.955854 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.955791 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:06.956138 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.956067 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "104430b4-2dbc-48f0-b96e-867018353f63" (UID: "104430b4-2dbc-48f0-b96e-867018353f63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:30:06.956261 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.956136 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104430b4-2dbc-48f0-b96e-867018353f63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "104430b4-2dbc-48f0-b96e-867018353f63" (UID: "104430b4-2dbc-48f0-b96e-867018353f63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:30:06.956333 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.956267 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-data" (OuterVolumeSpecName: "istio-data") pod "104430b4-2dbc-48f0-b96e-867018353f63" (UID: "104430b4-2dbc-48f0-b96e-867018353f63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:30:06.957575 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.957552 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/104430b4-2dbc-48f0-b96e-867018353f63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "104430b4-2dbc-48f0-b96e-867018353f63" (UID: "104430b4-2dbc-48f0-b96e-867018353f63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:30:06.957691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.957568 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-kube-api-access-d2n4s" (OuterVolumeSpecName: "kube-api-access-d2n4s") pod "104430b4-2dbc-48f0-b96e-867018353f63" (UID: "104430b4-2dbc-48f0-b96e-867018353f63"). InnerVolumeSpecName "kube-api-access-d2n4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:30:06.957691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.957570 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-istio-token" (OuterVolumeSpecName: "istio-token") pod "104430b4-2dbc-48f0-b96e-867018353f63" (UID: "104430b4-2dbc-48f0-b96e-867018353f63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:30:06.957775 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:06.957756 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "104430b4-2dbc-48f0-b96e-867018353f63" (UID: "104430b4-2dbc-48f0-b96e-867018353f63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:30:07.056730 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.056693 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:07.056730 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.056724 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:07.056730 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.056735 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2n4s\" (UniqueName: \"kubernetes.io/projected/104430b4-2dbc-48f0-b96e-867018353f63-kube-api-access-d2n4s\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:07.056974 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.056745 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/104430b4-2dbc-48f0-b96e-867018353f63-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:07.056974 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.056754 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:07.056974 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.056762 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/104430b4-2dbc-48f0-b96e-867018353f63-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:07.056974 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.056769 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/104430b4-2dbc-48f0-b96e-867018353f63-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:07.485260 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.485224 2562 generic.go:356] "Generic (PLEG): container finished" podID="104430b4-2dbc-48f0-b96e-867018353f63" containerID="00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d" exitCode=0 Jul 14 22:30:07.485464 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.485279 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" Jul 14 22:30:07.485464 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.485302 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" event={"ID":"104430b4-2dbc-48f0-b96e-867018353f63","Type":"ContainerDied","Data":"00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d"} Jul 14 22:30:07.485464 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.485356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h" event={"ID":"104430b4-2dbc-48f0-b96e-867018353f63","Type":"ContainerDied","Data":"c9de4faf2cdbcd99e2e27cd8470f28874570186713a787b3db48d693f73f5205"} Jul 14 22:30:07.485464 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.485378 2562 scope.go:117] "RemoveContainer" containerID="00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d" Jul 14 22:30:07.494865 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.494849 2562 scope.go:117] "RemoveContainer" containerID="16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018" Jul 14 22:30:07.503571 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.503552 2562 scope.go:117] "RemoveContainer" containerID="00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d" Jul 14 22:30:07.503819 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:30:07.503803 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d\": container with ID starting with 00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d not found: ID does not exist" containerID="00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d" Jul 14 22:30:07.503888 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.503825 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d"} err="failed to get container status \"00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d\": rpc error: code = NotFound desc = could not find container \"00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d\": container with ID starting with 00b39084d68ea017c60a394e1e7c7bc55d8a7e7606fcbb296e350ec53c2d5b1d not found: ID does not exist" Jul 14 22:30:07.503888 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.503840 2562 scope.go:117] "RemoveContainer" containerID="16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018" Jul 14 22:30:07.504065 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:30:07.504046 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018\": container with ID starting with 16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018 not found: ID does not exist" containerID="16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018" Jul 14 22:30:07.504124 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.504072 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018"} err="failed to get container status \"16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018\": rpc error: code = NotFound desc = could not find container \"16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018\": container with ID starting with 16c939ee90a4a8040ba0063cbd81932d7e5b461edc67a51caded548ad5d5c018 not found: ID does not exist" Jul 14 22:30:07.510453 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.510429 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h"] Jul 14 22:30:07.517302 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:07.517279 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6a0cfb-openshift-default-6b445c47d9-jnb4h"] Jul 14 22:30:08.809657 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:08.809622 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104430b4-2dbc-48f0-b96e-867018353f63" path="/var/lib/kubelet/pods/104430b4-2dbc-48f0-b96e-867018353f63/volumes" Jul 14 22:30:12.508809 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:12.508779 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9"] Jul 14 22:30:12.511139 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:12.511114 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4e43e-29734470-4h4w9"] Jul 14 22:30:12.810146 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:12.810109 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e7772c-387c-4f52-9967-0eff6af6e49a" path="/var/lib/kubelet/pods/30e7772c-387c-4f52-9967-0eff6af6e49a/volumes" Jul 14 22:30:27.860978 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:27.860944 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x"] Jul 14 22:30:27.861448 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:27.861187 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" containerID="cri-o://2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e" gracePeriod=30 Jul 14 22:30:29.132232 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.132197 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5"] Jul 14 22:30:29.132624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.132556 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" Jul 14 22:30:29.132624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.132567 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" Jul 14 22:30:29.132624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.132577 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" Jul 14 22:30:29.132624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.132582 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" Jul 14 22:30:29.132624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.132593 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e7772c-387c-4f52-9967-0eff6af6e49a" containerName="cleanup" Jul 14 22:30:29.132624 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.132599 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e7772c-387c-4f52-9967-0eff6af6e49a" containerName="cleanup" Jul 14 22:30:29.132821 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.132665 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" Jul 14 22:30:29.132821 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.132675 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e7772c-387c-4f52-9967-0eff6af6e49a" containerName="cleanup" Jul 14 22:30:29.135872 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.135849 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.141614 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.141588 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-de6a00-openshift-default-dockercfg-dtv2v\"" Jul 14 22:30:29.168541 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.168507 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5"] Jul 14 22:30:29.252963 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.252925 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-certs\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.253203 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.253062 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-socket\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.253203 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.253142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-credential-socket\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.253203 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.253175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-podinfo\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.253203 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.253198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-envoy\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.253429 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.253237 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-token\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.253429 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.253267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istiod-ca-cert\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.253429 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.253299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzjgb\" (UniqueName: \"kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-kube-api-access-wzjgb\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.253429 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.253413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-data\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354226 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-credential-socket\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354226 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-podinfo\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354497 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354248 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-envoy\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354497 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354264 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-token\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354497 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istiod-ca-cert\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354497 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzjgb\" (UniqueName: \"kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-kube-api-access-wzjgb\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354497 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-data\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354497 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-certs\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354497 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-socket\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354861 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354677 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-credential-socket\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354861 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354764 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-data\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.354861 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354803 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-certs\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.355060 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-socket\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.355060 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.354983 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istiod-ca-cert\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.356586 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.356566 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-envoy\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.356904 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.356887 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-podinfo\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.369577 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.369544 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-token\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.369759 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.369740 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzjgb\" (UniqueName: \"kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-kube-api-access-wzjgb\") pod \"e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.446415 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.446316 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:29.589756 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.589733 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5"] Jul 14 22:30:29.590934 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:30:29.590911 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4f5ed5_ce98_484c_a0cc_9073aac29cab.slice/crio-ec8cbb7019d57a00a0d090ca9d91b091c997ad542c3ec43336bcd6cc0d297d06 WatchSource:0}: Error finding container ec8cbb7019d57a00a0d090ca9d91b091c997ad542c3ec43336bcd6cc0d297d06: Status 404 returned error can't find the container with id ec8cbb7019d57a00a0d090ca9d91b091c997ad542c3ec43336bcd6cc0d297d06 Jul 14 22:30:29.592935 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.592902 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:29.592996 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.592971 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:29.593044 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.592999 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:29.618885 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:29.618858 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" event={"ID":"af4f5ed5-ce98-484c-a0cc-9073aac29cab","Type":"ContainerStarted","Data":"ec8cbb7019d57a00a0d090ca9d91b091c997ad542c3ec43336bcd6cc0d297d06"} Jul 14 22:30:30.626277 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:30.626243 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" event={"ID":"af4f5ed5-ce98-484c-a0cc-9073aac29cab","Type":"ContainerStarted","Data":"fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257"} Jul 14 22:30:30.656086 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:30.656017 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" podStartSLOduration=1.656002188 podStartE2EDuration="1.656002188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:30:30.653753739 +0000 UTC m=+2365.503223778" watchObservedRunningTime="2026-07-14 22:30:30.656002188 +0000 UTC m=+2365.505472229" Jul 14 22:30:31.447079 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:31.447041 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:31.451936 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:31.451911 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:31.631904 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:31.631874 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:31.632835 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:31.632817 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:33.127119 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.127052 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:30:33.297305 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297271 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-socket\") pod \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " Jul 14 22:30:33.297501 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297333 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istiod-ca-cert\") pod \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " Jul 14 22:30:33.297501 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297365 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-data\") pod \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " Jul 14 22:30:33.297501 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297382 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-envoy\") pod \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " Jul 14 22:30:33.297501 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297402 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-podinfo\") pod \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " Jul 14 22:30:33.297501 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297443 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-credential-socket\") pod \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " Jul 14 22:30:33.297501 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297471 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-certs\") pod \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " Jul 14 22:30:33.297826 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297529 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-token\") pod \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " Jul 14 22:30:33.297826 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297560 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rflkk\" (UniqueName: \"kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-kube-api-access-rflkk\") pod \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\" (UID: \"0794d30e-7f50-4c8e-84ab-06fbe68afe04\") " Jul 14 22:30:33.297826 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297607 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0794d30e-7f50-4c8e-84ab-06fbe68afe04" (UID: "0794d30e-7f50-4c8e-84ab-06fbe68afe04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:30:33.297989 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297825 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0794d30e-7f50-4c8e-84ab-06fbe68afe04" (UID: "0794d30e-7f50-4c8e-84ab-06fbe68afe04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:30:33.297989 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297843 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0794d30e-7f50-4c8e-84ab-06fbe68afe04" (UID: "0794d30e-7f50-4c8e-84ab-06fbe68afe04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:30:33.297989 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.297880 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:33.298221 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.298202 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0794d30e-7f50-4c8e-84ab-06fbe68afe04" (UID: "0794d30e-7f50-4c8e-84ab-06fbe68afe04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:30:33.298364 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.298343 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-data" (OuterVolumeSpecName: "istio-data") pod "0794d30e-7f50-4c8e-84ab-06fbe68afe04" (UID: "0794d30e-7f50-4c8e-84ab-06fbe68afe04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:30:33.300371 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.300348 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0794d30e-7f50-4c8e-84ab-06fbe68afe04" (UID: "0794d30e-7f50-4c8e-84ab-06fbe68afe04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:30:33.300371 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.300364 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-kube-api-access-rflkk" (OuterVolumeSpecName: "kube-api-access-rflkk") pod "0794d30e-7f50-4c8e-84ab-06fbe68afe04" (UID: "0794d30e-7f50-4c8e-84ab-06fbe68afe04"). InnerVolumeSpecName "kube-api-access-rflkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:30:33.300536 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.300389 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0794d30e-7f50-4c8e-84ab-06fbe68afe04" (UID: "0794d30e-7f50-4c8e-84ab-06fbe68afe04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:30:33.300536 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.300405 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-token" (OuterVolumeSpecName: "istio-token") pod "0794d30e-7f50-4c8e-84ab-06fbe68afe04" (UID: "0794d30e-7f50-4c8e-84ab-06fbe68afe04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:30:33.399310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.399214 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:33.399310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.399246 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:33.399310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.399257 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:33.399310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.399266 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:33.399310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.399275 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rflkk\" (UniqueName: \"kubernetes.io/projected/0794d30e-7f50-4c8e-84ab-06fbe68afe04-kube-api-access-rflkk\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:33.399310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.399283 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:33.399310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.399291 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:33.399310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.399299 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0794d30e-7f50-4c8e-84ab-06fbe68afe04-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:33.649070 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.648851 2562 generic.go:356] "Generic (PLEG): container finished" podID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerID="2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e" exitCode=0 Jul 14 22:30:33.649958 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.649875 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" Jul 14 22:30:33.650121 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.649867 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" event={"ID":"0794d30e-7f50-4c8e-84ab-06fbe68afe04","Type":"ContainerDied","Data":"2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e"} Jul 14 22:30:33.650121 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.650049 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x" event={"ID":"0794d30e-7f50-4c8e-84ab-06fbe68afe04","Type":"ContainerDied","Data":"f60c76aecd153c9ed0c48f954154e6d438a827387cd46c375bd926fbb8b69ae4"} Jul 14 22:30:33.650121 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.650086 2562 scope.go:117] "RemoveContainer" containerID="2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e" Jul 14 22:30:33.667811 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.667789 2562 scope.go:117] "RemoveContainer" containerID="4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa" Jul 14 22:30:33.676421 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.676401 2562 scope.go:117] "RemoveContainer" containerID="2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e" Jul 14 22:30:33.676684 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:30:33.676662 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e\": container with ID starting with 2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e not found: ID does not exist" containerID="2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e" Jul 14 22:30:33.676748 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.676692 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e"} err="failed to get container status \"2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e\": rpc error: code = NotFound desc = could not find container \"2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e\": container with ID starting with 2161c05d0a5f04f807cec3ce435ffd2920607da6b32487f14a93ba69eabe7b0e not found: ID does not exist" Jul 14 22:30:33.676748 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.676710 2562 scope.go:117] "RemoveContainer" containerID="4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa" Jul 14 22:30:33.676965 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:30:33.676934 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa\": container with ID starting with 4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa not found: ID does not exist" containerID="4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa" Jul 14 22:30:33.677006 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.676969 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa"} err="failed to get container status \"4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa\": rpc error: code = NotFound desc = could not find container \"4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa\": container with ID starting with 4cb14d39c89be7c17643f37b1964053625937b00a09dd5a35d18f68b7b83e4fa not found: ID does not exist" Jul 14 22:30:33.690105 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.690077 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x"] Jul 14 22:30:33.692513 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:33.692492 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b4e43e-openshift-default-dc67544d9-wqs5x"] Jul 14 22:30:34.810141 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:34.810096 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" path="/var/lib/kubelet/pods/0794d30e-7f50-4c8e-84ab-06fbe68afe04/volumes" Jul 14 22:30:38.213433 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.212743 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967"] Jul 14 22:30:38.213433 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.213319 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" Jul 14 22:30:38.213433 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.213340 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" Jul 14 22:30:38.213433 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.213363 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" Jul 14 22:30:38.213433 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.213373 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" Jul 14 22:30:38.214170 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.213473 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" Jul 14 22:30:38.214170 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.213487 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="104430b4-2dbc-48f0-b96e-867018353f63" containerName="istio-proxy" Jul 14 22:30:38.218643 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.218592 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.222271 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.222246 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dc25fd-openshift-default-dockercfg-7pp55\"" Jul 14 22:30:38.227530 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.227507 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967"] Jul 14 22:30:38.239353 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.239328 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-data\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.239478 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.239387 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-socket\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.239478 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.239407 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istiod-ca-cert\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.239478 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.239426 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-certs\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.239587 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.239482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-podinfo\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.239587 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.239504 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-token\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.239587 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.239538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbq4\" (UniqueName: \"kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-kube-api-access-gbbq4\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.239587 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.239568 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-credential-socket\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.239713 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.239619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-envoy\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.340789 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.340740 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-envoy\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.340813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-data\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.340873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-socket\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.340892 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istiod-ca-cert\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.340912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-certs\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.340952 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-podinfo\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.340977 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-token\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341004 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.340993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbq4\" (UniqueName: \"kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-kube-api-access-gbbq4\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341428 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.341011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-credential-socket\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341428 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.341304 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-data\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341428 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.341336 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-socket\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341428 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.341394 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-credential-socket\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341640 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.341588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-certs\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.341953 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.341932 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istiod-ca-cert\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.343382 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.343364 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-envoy\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.343659 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.343642 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-podinfo\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.351582 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.351562 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-token\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.352242 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.352221 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbq4\" (UniqueName: \"kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-kube-api-access-gbbq4\") pod \"e2e-api-b-dc25fd-openshift-default-66767574b9-g8967\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.531327 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.531294 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:38.708809 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.708692 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967"] Jul 14 22:30:38.717115 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.717078 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:38.717223 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.717160 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:38.717223 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:38.717203 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:39.692975 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:39.692932 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" event={"ID":"d4c381e1-8e83-45a7-bf11-9b29889f0efc","Type":"ContainerStarted","Data":"e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c"} Jul 14 22:30:39.693487 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:39.693004 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" event={"ID":"d4c381e1-8e83-45a7-bf11-9b29889f0efc","Type":"ContainerStarted","Data":"7fdaacc2472dc5df4714550d55dd6dcae7c4ecb50edf42ec034ceaaced0410fe"} Jul 14 22:30:39.719084 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:39.719004 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" podStartSLOduration=1.7189876480000001 podStartE2EDuration="1.718987648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:30:39.718591337 +0000 UTC m=+2374.568061398" watchObservedRunningTime="2026-07-14 22:30:39.718987648 +0000 UTC m=+2374.568457687" Jul 14 22:30:40.532165 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:40.532106 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:40.537175 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:40.537144 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:40.698871 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:40.698840 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:40.699975 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:40.699953 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:30:43.717201 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:43.717168 2562 generic.go:356] "Generic (PLEG): container finished" podID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerID="fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257" exitCode=0 Jul 14 22:30:43.717647 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:43.717243 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" event={"ID":"af4f5ed5-ce98-484c-a0cc-9073aac29cab","Type":"ContainerDied","Data":"fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257"} Jul 14 22:30:43.717733 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:43.717703 2562 scope.go:117] "RemoveContainer" containerID="fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257" Jul 14 22:30:43.718962 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:43.718931 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:43.719052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:43.719009 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:43.719099 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:43.719076 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:44.725685 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:44.725650 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" event={"ID":"af4f5ed5-ce98-484c-a0cc-9073aac29cab","Type":"ContainerStarted","Data":"634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd"} Jul 14 22:30:45.446862 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:45.446826 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:45.448085 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:45.448058 2562 patch_prober.go:28] interesting pod/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 14 22:30:45.448221 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:45.448106 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 14 22:30:46.447399 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:46.447363 2562 patch_prober.go:28] interesting pod/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 14 22:30:46.447786 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:46.447421 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 14 22:30:46.632096 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:46.632059 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:47.447427 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:47.447391 2562 patch_prober.go:28] interesting pod/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 14 22:30:47.447834 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:47.447452 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 14 22:30:48.447701 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:48.447663 2562 patch_prober.go:28] interesting pod/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 14 22:30:48.448116 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:48.447721 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 14 22:30:49.451387 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:49.451358 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:49.452548 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:49.452526 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:30:54.415615 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:54.415570 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr"] Jul 14 22:30:54.418541 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:54.418514 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0794d30e-7f50-4c8e-84ab-06fbe68afe04" containerName="istio-proxy" Jul 14 22:30:54.422226 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:54.422202 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" Jul 14 22:30:54.425395 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:54.425367 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:30:54.425644 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:54.425393 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:30:54.427627 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:54.427600 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr"] Jul 14 22:30:54.499050 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:54.498561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e87752-6ad2-4410-bf7f-2dab6345c696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr\" (UID: \"e9e87752-6ad2-4410-bf7f-2dab6345c696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" Jul 14 22:30:54.599677 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:54.599623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e87752-6ad2-4410-bf7f-2dab6345c696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr\" (UID: \"e9e87752-6ad2-4410-bf7f-2dab6345c696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" Jul 14 22:30:54.600677 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:54.600640 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e87752-6ad2-4410-bf7f-2dab6345c696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr\" (UID: \"e9e87752-6ad2-4410-bf7f-2dab6345c696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" Jul 14 22:30:54.741933 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:54.741838 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" Jul 14 22:30:55.130189 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:55.130158 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr"] Jul 14 22:30:55.132094 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:30:55.132060 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e87752_6ad2_4410_bf7f_2dab6345c696.slice/crio-f012a6516152556543fa221a91f7db411d9ae4200d739b2772271c89f971fac4 WatchSource:0}: Error finding container f012a6516152556543fa221a91f7db411d9ae4200d739b2772271c89f971fac4: Status 404 returned error can't find the container with id f012a6516152556543fa221a91f7db411d9ae4200d739b2772271c89f971fac4 Jul 14 22:30:55.808832 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:55.808786 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" event={"ID":"e9e87752-6ad2-4410-bf7f-2dab6345c696","Type":"ContainerStarted","Data":"382c3bde99d3250dc91883731ea4b9d57440c5b4ce0f8925696988e933127ed2"} Jul 14 22:30:55.809376 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:55.808841 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" event={"ID":"e9e87752-6ad2-4410-bf7f-2dab6345c696","Type":"ContainerStarted","Data":"f012a6516152556543fa221a91f7db411d9ae4200d739b2772271c89f971fac4"} Jul 14 22:30:55.826216 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:55.826165 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" podStartSLOduration=1.8261460600000001 podStartE2EDuration="1.82614606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:30:55.826051259 +0000 UTC m=+2390.675521292" watchObservedRunningTime="2026-07-14 22:30:55.82614606 +0000 UTC m=+2390.675616100" Jul 14 22:30:56.816387 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:56.816299 2562 generic.go:356] "Generic (PLEG): container finished" podID="e9e87752-6ad2-4410-bf7f-2dab6345c696" containerID="382c3bde99d3250dc91883731ea4b9d57440c5b4ce0f8925696988e933127ed2" exitCode=0 Jul 14 22:30:56.816742 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:56.816379 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" event={"ID":"e9e87752-6ad2-4410-bf7f-2dab6345c696","Type":"ContainerDied","Data":"382c3bde99d3250dc91883731ea4b9d57440c5b4ce0f8925696988e933127ed2"} Jul 14 22:30:57.967295 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:57.967270 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" Jul 14 22:30:58.035231 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:58.035194 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e87752-6ad2-4410-bf7f-2dab6345c696-maas-api-service-ca\") pod \"e9e87752-6ad2-4410-bf7f-2dab6345c696\" (UID: \"e9e87752-6ad2-4410-bf7f-2dab6345c696\") " Jul 14 22:30:58.035583 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:58.035560 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e87752-6ad2-4410-bf7f-2dab6345c696-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9e87752-6ad2-4410-bf7f-2dab6345c696" (UID: "e9e87752-6ad2-4410-bf7f-2dab6345c696"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:30:58.136543 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:58.136456 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e87752-6ad2-4410-bf7f-2dab6345c696-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:30:58.830143 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:58.830117 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" Jul 14 22:30:58.830335 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:58.830148 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr" event={"ID":"e9e87752-6ad2-4410-bf7f-2dab6345c696","Type":"ContainerDied","Data":"f012a6516152556543fa221a91f7db411d9ae4200d739b2772271c89f971fac4"} Jul 14 22:30:58.830335 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:58.830220 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f012a6516152556543fa221a91f7db411d9ae4200d739b2772271c89f971fac4" Jul 14 22:30:59.836874 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:59.836841 2562 generic.go:356] "Generic (PLEG): container finished" podID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerID="e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c" exitCode=0 Jul 14 22:30:59.837432 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:59.836921 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" event={"ID":"d4c381e1-8e83-45a7-bf11-9b29889f0efc","Type":"ContainerDied","Data":"e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c"} Jul 14 22:30:59.837501 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:59.837467 2562 scope.go:117] "RemoveContainer" containerID="e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c" Jul 14 22:30:59.838490 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:59.838461 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:59.838558 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:59.838528 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:30:59.838600 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:30:59.838583 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:31:00.844943 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:00.844907 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" event={"ID":"d4c381e1-8e83-45a7-bf11-9b29889f0efc","Type":"ContainerStarted","Data":"bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58"} Jul 14 22:31:01.532393 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:01.532352 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:31:01.532600 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:01.532579 2562 patch_prober.go:28] interesting pod/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 14 22:31:01.532683 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:01.532647 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 14 22:31:02.532560 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:02.532524 2562 patch_prober.go:28] interesting pod/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 14 22:31:02.532953 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:02.532582 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 14 22:31:03.531821 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:03.531781 2562 patch_prober.go:28] interesting pod/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 14 22:31:03.532071 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:03.531842 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 14 22:31:04.531981 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:04.531941 2562 patch_prober.go:28] interesting pod/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 14 22:31:04.532493 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:04.531996 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 14 22:31:05.535050 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:05.534994 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:31:05.535526 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:05.535352 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:31:05.536100 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:05.536079 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:31:20.224008 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:20.223969 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5"] Jul 14 22:31:20.224433 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:20.224308 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" containerID="cri-o://634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd" gracePeriod=30 Jul 14 22:31:25.480041 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.479995 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:31:25.598396 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598355 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-socket\") pod \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " Jul 14 22:31:25.598616 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598415 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzjgb\" (UniqueName: \"kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-kube-api-access-wzjgb\") pod \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " Jul 14 22:31:25.598616 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598449 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-credential-socket\") pod \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " Jul 14 22:31:25.598616 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598496 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-podinfo\") pod \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " Jul 14 22:31:25.598616 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598520 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-token\") pod \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " Jul 14 22:31:25.598616 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598553 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-certs\") pod \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " Jul 14 22:31:25.598616 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598584 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istiod-ca-cert\") pod \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " Jul 14 22:31:25.598616 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598612 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-envoy\") pod \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " Jul 14 22:31:25.599002 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598615 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af4f5ed5-ce98-484c-a0cc-9073aac29cab" (UID: "af4f5ed5-ce98-484c-a0cc-9073aac29cab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:31:25.599002 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598699 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-data\") pod \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\" (UID: \"af4f5ed5-ce98-484c-a0cc-9073aac29cab\") " Jul 14 22:31:25.599002 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598764 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af4f5ed5-ce98-484c-a0cc-9073aac29cab" (UID: "af4f5ed5-ce98-484c-a0cc-9073aac29cab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:31:25.599002 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598864 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af4f5ed5-ce98-484c-a0cc-9073aac29cab" (UID: "af4f5ed5-ce98-484c-a0cc-9073aac29cab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:31:25.599002 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.598934 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af4f5ed5-ce98-484c-a0cc-9073aac29cab" (UID: "af4f5ed5-ce98-484c-a0cc-9073aac29cab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:31:25.599306 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.599189 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:25.599306 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.599215 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:25.599306 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.599229 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:25.599306 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.599243 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:25.599677 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.599654 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-data" (OuterVolumeSpecName: "istio-data") pod "af4f5ed5-ce98-484c-a0cc-9073aac29cab" (UID: "af4f5ed5-ce98-484c-a0cc-9073aac29cab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:31:25.600794 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.600764 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af4f5ed5-ce98-484c-a0cc-9073aac29cab" (UID: "af4f5ed5-ce98-484c-a0cc-9073aac29cab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:31:25.600897 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.600806 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-kube-api-access-wzjgb" (OuterVolumeSpecName: "kube-api-access-wzjgb") pod "af4f5ed5-ce98-484c-a0cc-9073aac29cab" (UID: "af4f5ed5-ce98-484c-a0cc-9073aac29cab"). InnerVolumeSpecName "kube-api-access-wzjgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:31:25.601127 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.601095 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af4f5ed5-ce98-484c-a0cc-9073aac29cab" (UID: "af4f5ed5-ce98-484c-a0cc-9073aac29cab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:31:25.601227 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.601146 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-token" (OuterVolumeSpecName: "istio-token") pod "af4f5ed5-ce98-484c-a0cc-9073aac29cab" (UID: "af4f5ed5-ce98-484c-a0cc-9073aac29cab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:31:25.700529 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.700488 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzjgb\" (UniqueName: \"kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-kube-api-access-wzjgb\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:25.700529 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.700524 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:25.700529 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.700538 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:25.700805 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.700548 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:25.700805 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.700555 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af4f5ed5-ce98-484c-a0cc-9073aac29cab-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:25.988074 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.987963 2562 generic.go:356] "Generic (PLEG): container finished" podID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerID="634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd" exitCode=0 Jul 14 22:31:25.988074 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.988047 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" Jul 14 22:31:25.988253 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.988065 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" event={"ID":"af4f5ed5-ce98-484c-a0cc-9073aac29cab","Type":"ContainerDied","Data":"634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd"} Jul 14 22:31:25.988253 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.988131 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5" event={"ID":"af4f5ed5-ce98-484c-a0cc-9073aac29cab","Type":"ContainerDied","Data":"ec8cbb7019d57a00a0d090ca9d91b091c997ad542c3ec43336bcd6cc0d297d06"} Jul 14 22:31:25.988253 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.988156 2562 scope.go:117] "RemoveContainer" containerID="634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd" Jul 14 22:31:25.997940 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:25.997922 2562 scope.go:117] "RemoveContainer" containerID="fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257" Jul 14 22:31:26.006470 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:26.006449 2562 scope.go:117] "RemoveContainer" containerID="634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd" Jul 14 22:31:26.006724 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:31:26.006702 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd\": container with ID starting with 634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd not found: ID does not exist" containerID="634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd" Jul 14 22:31:26.006791 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:26.006734 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd"} err="failed to get container status \"634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd\": rpc error: code = NotFound desc = could not find container \"634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd\": container with ID starting with 634c95342c2e11907b1eaf27ba8bb91cc38c7aff6ef974fa933a7430afeb7fcd not found: ID does not exist" Jul 14 22:31:26.006791 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:26.006752 2562 scope.go:117] "RemoveContainer" containerID="fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257" Jul 14 22:31:26.007008 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:31:26.006990 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257\": container with ID starting with fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257 not found: ID does not exist" containerID="fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257" Jul 14 22:31:26.007069 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:26.007015 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257"} err="failed to get container status \"fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257\": rpc error: code = NotFound desc = could not find container \"fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257\": container with ID starting with fb47a57e3b857f49e7ff109c26c5343334cac97371fa0d037c7ca4c2f4c81257 not found: ID does not exist" Jul 14 22:31:26.011249 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:26.011228 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5"] Jul 14 22:31:26.015301 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:26.015281 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-de6a00-openshift-default-6bdbdf847-mxqc5"] Jul 14 22:31:26.820076 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:26.820043 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" path="/var/lib/kubelet/pods/af4f5ed5-ce98-484c-a0cc-9073aac29cab/volumes" Jul 14 22:31:46.460705 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:46.460665 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967"] Jul 14 22:31:46.461286 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:46.460879 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" containerID="cri-o://bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58" gracePeriod=30 Jul 14 22:31:47.525143 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.525112 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p"] Jul 14 22:31:47.525592 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.525533 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" Jul 14 22:31:47.525592 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.525546 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" Jul 14 22:31:47.525592 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.525555 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" Jul 14 22:31:47.525592 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.525562 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" Jul 14 22:31:47.525592 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.525574 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e87752-6ad2-4410-bf7f-2dab6345c696" containerName="revoke-keys" Jul 14 22:31:47.525592 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.525580 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e87752-6ad2-4410-bf7f-2dab6345c696" containerName="revoke-keys" Jul 14 22:31:47.525789 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.525648 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e87752-6ad2-4410-bf7f-2dab6345c696" containerName="revoke-keys" Jul 14 22:31:47.525789 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.525658 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" Jul 14 22:31:47.525789 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.525667 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4f5ed5-ce98-484c-a0cc-9073aac29cab" containerName="istio-proxy" Jul 14 22:31:47.528666 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.528647 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.532609 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.532587 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-fdfd38-openshift-default-dockercfg-hbwvx\"" Jul 14 22:31:47.547310 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.547279 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p"] Jul 14 22:31:47.707879 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.707837 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-data\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.707879 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.707881 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-envoy\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.708148 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.707921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qlcl\" (UniqueName: \"kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-kube-api-access-6qlcl\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.708148 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.707944 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-credential-socket\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.708148 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.707970 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-socket\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.708148 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.708085 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-certs\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.708148 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.708122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-istio-token\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.708315 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.708184 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a786c041-5700-4310-9c47-965c43a3e7dd-istiod-ca-cert\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.708315 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.708205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a786c041-5700-4310-9c47-965c43a3e7dd-istio-podinfo\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.808756 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.808665 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-istio-token\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.808756 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.808717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a786c041-5700-4310-9c47-965c43a3e7dd-istiod-ca-cert\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.808756 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.808736 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a786c041-5700-4310-9c47-965c43a3e7dd-istio-podinfo\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.808756 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.808755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-data\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.809116 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.808776 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-envoy\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.809116 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.808931 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qlcl\" (UniqueName: \"kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-kube-api-access-6qlcl\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.809116 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.808981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-credential-socket\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.809116 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.809080 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-socket\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.809332 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.809145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-certs\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.809332 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.809250 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-data\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.809444 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.809366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-credential-socket\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.809444 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.809430 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-socket\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.809555 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.809534 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-certs\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.809555 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.809545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a786c041-5700-4310-9c47-965c43a3e7dd-istiod-ca-cert\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.811168 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.811147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-envoy\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.811435 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.811416 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a786c041-5700-4310-9c47-965c43a3e7dd-istio-podinfo\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.824504 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.824474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-istio-token\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.824746 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.824729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qlcl\" (UniqueName: \"kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-kube-api-access-6qlcl\") pod \"e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.839350 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.839327 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:47.998957 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:47.998926 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p"] Jul 14 22:31:48.000009 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:31:47.999980 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda786c041_5700_4310_9c47_965c43a3e7dd.slice/crio-8de9607d31d89abffba683bab62d02447886aad0315b902802a2a541affeba1b WatchSource:0}: Error finding container 8de9607d31d89abffba683bab62d02447886aad0315b902802a2a541affeba1b: Status 404 returned error can't find the container with id 8de9607d31d89abffba683bab62d02447886aad0315b902802a2a541affeba1b Jul 14 22:31:48.002206 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:48.002171 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:31:48.002294 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:48.002240 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:31:48.002294 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:48.002271 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:31:48.125865 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:48.125822 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" event={"ID":"a786c041-5700-4310-9c47-965c43a3e7dd","Type":"ContainerStarted","Data":"ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5"} Jul 14 22:31:48.126067 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:48.125888 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" event={"ID":"a786c041-5700-4310-9c47-965c43a3e7dd","Type":"ContainerStarted","Data":"8de9607d31d89abffba683bab62d02447886aad0315b902802a2a541affeba1b"} Jul 14 22:31:48.150694 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:48.150629 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" podStartSLOduration=1.150605773 podStartE2EDuration="1.150605773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:31:48.147613911 +0000 UTC m=+2442.997083961" watchObservedRunningTime="2026-07-14 22:31:48.150605773 +0000 UTC m=+2443.000075813" Jul 14 22:31:48.840347 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:48.840312 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:49.844858 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:49.844829 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:50.138633 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:50.138540 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:50.139509 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:50.139491 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:31:50.537820 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:50.537785 2562 patch_prober.go:28] interesting pod/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:31:50.538061 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:50.537848 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:31:51.719600 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.719576 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:31:51.846955 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.846916 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istiod-ca-cert\") pod \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " Jul 14 22:31:51.847184 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847001 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-certs\") pod \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " Jul 14 22:31:51.847184 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847017 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-envoy\") pod \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " Jul 14 22:31:51.847184 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847084 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-socket\") pod \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " Jul 14 22:31:51.847184 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847102 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbbq4\" (UniqueName: \"kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-kube-api-access-gbbq4\") pod \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " Jul 14 22:31:51.847184 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847124 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-data\") pod \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " Jul 14 22:31:51.847460 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847242 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-token\") pod \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " Jul 14 22:31:51.847460 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847272 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-credential-socket\") pod \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " Jul 14 22:31:51.847460 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847317 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-podinfo\") pod \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\" (UID: \"d4c381e1-8e83-45a7-bf11-9b29889f0efc\") " Jul 14 22:31:51.847460 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847365 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4c381e1-8e83-45a7-bf11-9b29889f0efc" (UID: "d4c381e1-8e83-45a7-bf11-9b29889f0efc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:31:51.847675 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847462 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4c381e1-8e83-45a7-bf11-9b29889f0efc" (UID: "d4c381e1-8e83-45a7-bf11-9b29889f0efc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:31:51.847675 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847471 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4c381e1-8e83-45a7-bf11-9b29889f0efc" (UID: "d4c381e1-8e83-45a7-bf11-9b29889f0efc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:31:51.847675 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847583 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4c381e1-8e83-45a7-bf11-9b29889f0efc" (UID: "d4c381e1-8e83-45a7-bf11-9b29889f0efc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:31:51.847829 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847683 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:51.847829 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847704 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:51.847829 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847718 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:51.847829 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.847735 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:51.848362 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.848334 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4c381e1-8e83-45a7-bf11-9b29889f0efc" (UID: "d4c381e1-8e83-45a7-bf11-9b29889f0efc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:31:51.849483 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.849459 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4c381e1-8e83-45a7-bf11-9b29889f0efc" (UID: "d4c381e1-8e83-45a7-bf11-9b29889f0efc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:31:51.849598 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.849539 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-kube-api-access-gbbq4" (OuterVolumeSpecName: "kube-api-access-gbbq4") pod "d4c381e1-8e83-45a7-bf11-9b29889f0efc" (UID: "d4c381e1-8e83-45a7-bf11-9b29889f0efc"). InnerVolumeSpecName "kube-api-access-gbbq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:31:51.849598 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.849578 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4c381e1-8e83-45a7-bf11-9b29889f0efc" (UID: "d4c381e1-8e83-45a7-bf11-9b29889f0efc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:31:51.849598 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.849590 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4c381e1-8e83-45a7-bf11-9b29889f0efc" (UID: "d4c381e1-8e83-45a7-bf11-9b29889f0efc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:31:51.948506 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.948464 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:51.948506 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.948498 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:51.948506 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.948509 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbbq4\" (UniqueName: \"kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-kube-api-access-gbbq4\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:51.948506 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.948517 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:51.948836 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:51.948527 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c381e1-8e83-45a7-bf11-9b29889f0efc-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:31:52.153543 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.153453 2562 generic.go:356] "Generic (PLEG): container finished" podID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerID="bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58" exitCode=0 Jul 14 22:31:52.153543 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.153498 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" Jul 14 22:31:52.153731 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.153540 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" event={"ID":"d4c381e1-8e83-45a7-bf11-9b29889f0efc","Type":"ContainerDied","Data":"bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58"} Jul 14 22:31:52.153731 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.153591 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967" event={"ID":"d4c381e1-8e83-45a7-bf11-9b29889f0efc","Type":"ContainerDied","Data":"7fdaacc2472dc5df4714550d55dd6dcae7c4ecb50edf42ec034ceaaced0410fe"} Jul 14 22:31:52.153731 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.153615 2562 scope.go:117] "RemoveContainer" containerID="bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58" Jul 14 22:31:52.163792 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.163775 2562 scope.go:117] "RemoveContainer" containerID="e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c" Jul 14 22:31:52.172100 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.172083 2562 scope.go:117] "RemoveContainer" containerID="bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58" Jul 14 22:31:52.172344 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:31:52.172325 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58\": container with ID starting with bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58 not found: ID does not exist" containerID="bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58" Jul 14 22:31:52.172401 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.172354 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58"} err="failed to get container status \"bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58\": rpc error: code = NotFound desc = could not find container \"bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58\": container with ID starting with bc3a99926f65f1057857414203e939b2b9cd0f66b15793fe804066cc8c66bd58 not found: ID does not exist" Jul 14 22:31:52.172401 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.172373 2562 scope.go:117] "RemoveContainer" containerID="e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c" Jul 14 22:31:52.172595 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:31:52.172578 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c\": container with ID starting with e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c not found: ID does not exist" containerID="e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c" Jul 14 22:31:52.172637 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.172601 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c"} err="failed to get container status \"e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c\": rpc error: code = NotFound desc = could not find container \"e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c\": container with ID starting with e1d9fa6d8722f2c68901e160eb6f69ca490a13535beb14676065cfb9e6d1258c not found: ID does not exist" Jul 14 22:31:52.176817 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.176794 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967"] Jul 14 22:31:52.180771 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.180750 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dc25fd-openshift-default-66767574b9-g8967"] Jul 14 22:31:52.809902 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:52.809866 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" path="/var/lib/kubelet/pods/d4c381e1-8e83-45a7-bf11-9b29889f0efc/volumes" Jul 14 22:31:55.233529 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.233494 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75"] Jul 14 22:31:55.233925 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.233862 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" Jul 14 22:31:55.233925 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.233874 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" Jul 14 22:31:55.233925 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.233890 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" Jul 14 22:31:55.233925 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.233896 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" Jul 14 22:31:55.234086 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.233957 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" Jul 14 22:31:55.234086 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.233964 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4c381e1-8e83-45a7-bf11-9b29889f0efc" containerName="istio-proxy" Jul 14 22:31:55.238737 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.238716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.241252 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.241230 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-32d7b3-openshift-default-dockercfg-fhlst\"" Jul 14 22:31:55.249274 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.249247 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75"] Jul 14 22:31:55.284200 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.284157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-credential-socket\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.284200 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.284205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9pb\" (UniqueName: \"kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-kube-api-access-6f9pb\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.284435 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.284227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-socket\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.284435 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.284250 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-envoy\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.284435 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.284276 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-podinfo\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.284435 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.284385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-data\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.284435 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.284421 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-token\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.284605 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.284460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-certs\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.284605 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.284477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istiod-ca-cert\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385080 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-credential-socket\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385242 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385109 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9pb\" (UniqueName: \"kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-kube-api-access-6f9pb\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385242 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-socket\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385242 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-envoy\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385242 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385221 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-podinfo\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385427 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-data\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385505 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-token\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385505 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385449 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-credential-socket\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385505 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385484 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-certs\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385505 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385490 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-socket\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385505 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istiod-ca-cert\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385809 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385735 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-data\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.385854 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.385809 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-certs\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.386143 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.386119 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istiod-ca-cert\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.388019 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.387994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-envoy\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.388341 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.388320 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-podinfo\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.398357 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.398335 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-token\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.402734 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.402711 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9pb\" (UniqueName: \"kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-kube-api-access-6f9pb\") pod \"e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.551740 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.551699 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:55.700604 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.700575 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75"] Jul 14 22:31:55.701855 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:31:55.701825 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e24446_e3da_4f16_bb49_185ee5cd2ab9.slice/crio-11d4d40911cfa603c5dca7a0ce4eced688ffe8e297556dffad0441b5e074a9ff WatchSource:0}: Error finding container 11d4d40911cfa603c5dca7a0ce4eced688ffe8e297556dffad0441b5e074a9ff: Status 404 returned error can't find the container with id 11d4d40911cfa603c5dca7a0ce4eced688ffe8e297556dffad0441b5e074a9ff Jul 14 22:31:55.703899 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.703863 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:31:55.703973 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.703930 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:31:55.703973 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:55.703961 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:31:56.179637 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:56.179601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" event={"ID":"42e24446-e3da-4f16-bb49-185ee5cd2ab9","Type":"ContainerStarted","Data":"e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209"} Jul 14 22:31:56.179823 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:56.179662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" event={"ID":"42e24446-e3da-4f16-bb49-185ee5cd2ab9","Type":"ContainerStarted","Data":"11d4d40911cfa603c5dca7a0ce4eced688ffe8e297556dffad0441b5e074a9ff"} Jul 14 22:31:56.205623 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:56.205575 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" podStartSLOduration=1.205557061 podStartE2EDuration="1.205557061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:31:56.203931074 +0000 UTC m=+2451.053401132" watchObservedRunningTime="2026-07-14 22:31:56.205557061 +0000 UTC m=+2451.055027099" Jul 14 22:31:56.552190 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:56.552162 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:56.557702 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:56.557676 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:57.186528 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:57.186496 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:31:57.187873 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:31:57.187851 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:32:05.140111 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:05.140078 2562 patch_prober.go:28] interesting pod/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 14 22:32:05.140514 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:05.140135 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 14 22:32:05.246415 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:05.246380 2562 generic.go:356] "Generic (PLEG): container finished" podID="a786c041-5700-4310-9c47-965c43a3e7dd" containerID="ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5" exitCode=0 Jul 14 22:32:05.246608 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:05.246457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" event={"ID":"a786c041-5700-4310-9c47-965c43a3e7dd","Type":"ContainerDied","Data":"ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5"} Jul 14 22:32:05.246973 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:05.246958 2562 scope.go:117] "RemoveContainer" containerID="ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5" Jul 14 22:32:05.248120 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:05.248087 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:32:05.248252 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:05.248170 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:32:05.248252 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:05.248220 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:32:06.255784 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:06.255744 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" event={"ID":"a786c041-5700-4310-9c47-965c43a3e7dd","Type":"ContainerStarted","Data":"daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411"} Jul 14 22:32:06.840400 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:06.840352 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:32:06.840679 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:06.840655 2562 patch_prober.go:28] interesting pod/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 14 22:32:06.840772 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:06.840709 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 14 22:32:07.839850 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:07.839817 2562 patch_prober.go:28] interesting pod/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 14 22:32:07.840263 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:07.839871 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 14 22:32:08.840044 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:08.839988 2562 patch_prober.go:28] interesting pod/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 14 22:32:08.840573 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:08.840090 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 14 22:32:09.840124 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:09.840077 2562 patch_prober.go:28] interesting pod/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 14 22:32:09.840551 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:09.840146 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 14 22:32:10.844564 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:10.844533 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:32:10.845858 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:10.845013 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:32:10.845858 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:10.845749 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:32:11.287631 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:11.287594 2562 generic.go:356] "Generic (PLEG): container finished" podID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerID="e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209" exitCode=0 Jul 14 22:32:11.287805 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:11.287666 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" event={"ID":"42e24446-e3da-4f16-bb49-185ee5cd2ab9","Type":"ContainerDied","Data":"e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209"} Jul 14 22:32:11.288109 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:11.288089 2562 scope.go:117] "RemoveContainer" containerID="e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209" Jul 14 22:32:11.289069 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:11.289044 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:32:11.289128 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:11.289113 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:32:11.289169 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:11.289145 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 22:32:12.186547 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:12.186509 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:32:12.295759 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:12.295721 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" event={"ID":"42e24446-e3da-4f16-bb49-185ee5cd2ab9","Type":"ContainerStarted","Data":"bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98"} Jul 14 22:32:12.552099 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:12.552061 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:32:12.553298 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:12.553269 2562 patch_prober.go:28] interesting pod/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 14 22:32:12.553417 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:12.553322 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 14 22:32:13.552305 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:13.552268 2562 patch_prober.go:28] interesting pod/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 14 22:32:13.552725 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:13.552323 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 14 22:32:14.552066 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:14.552005 2562 patch_prober.go:28] interesting pod/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 14 22:32:14.552303 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:14.552081 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 14 22:32:15.552817 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:15.552779 2562 patch_prober.go:28] interesting pod/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 14 22:32:15.553222 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:15.552843 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 14 22:32:16.566688 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:16.566655 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:32:16.567282 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:16.567247 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:32:16.568237 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:32:16.568216 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:36:07.151136 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:36:07.151055 2562 scope.go:117] "RemoveContainer" containerID="47a8649ad0c5c8f79dbffb542013fba335b024af0f87978d18528bdee68a9821" Jul 14 22:37:46.928794 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:46.928690 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc"] Jul 14 22:37:46.932555 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:46.932529 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc" Jul 14 22:37:46.935277 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:46.935249 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4jlcq\"" Jul 14 22:37:46.935435 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:46.935317 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:37:46.936898 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:46.936856 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc"] Jul 14 22:37:47.055132 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:47.055089 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/081cd25d-f9ef-4900-bf3a-e5e19a7b67e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc\" (UID: \"081cd25d-f9ef-4900-bf3a-e5e19a7b67e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc" Jul 14 22:37:47.156169 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:47.156133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/081cd25d-f9ef-4900-bf3a-e5e19a7b67e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc\" (UID: \"081cd25d-f9ef-4900-bf3a-e5e19a7b67e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc" Jul 14 22:37:47.156809 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:47.156786 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/081cd25d-f9ef-4900-bf3a-e5e19a7b67e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc\" (UID: \"081cd25d-f9ef-4900-bf3a-e5e19a7b67e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc" Jul 14 22:37:47.244972 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:47.244879 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc" Jul 14 22:37:47.386496 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:47.386473 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc"] Jul 14 22:37:47.388302 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:37:47.388277 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081cd25d_f9ef_4900_bf3a_e5e19a7b67e8.slice/crio-9407ad695affd72a91ddb2f9375252d6aaec9c9d3947c19f6ae81ee25bcb05ed WatchSource:0}: Error finding container 9407ad695affd72a91ddb2f9375252d6aaec9c9d3947c19f6ae81ee25bcb05ed: Status 404 returned error can't find the container with id 9407ad695affd72a91ddb2f9375252d6aaec9c9d3947c19f6ae81ee25bcb05ed Jul 14 22:37:48.225931 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:48.225887 2562 generic.go:356] "Generic (PLEG): container finished" podID="081cd25d-f9ef-4900-bf3a-e5e19a7b67e8" containerID="32bd925da22ac09dbc248973233b233e7f077241f1d6c99bda4f43d9bc45ae09" exitCode=0 Jul 14 22:37:48.226385 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:48.225928 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc" event={"ID":"081cd25d-f9ef-4900-bf3a-e5e19a7b67e8","Type":"ContainerDied","Data":"32bd925da22ac09dbc248973233b233e7f077241f1d6c99bda4f43d9bc45ae09"} Jul 14 22:37:48.226385 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:48.225982 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc" event={"ID":"081cd25d-f9ef-4900-bf3a-e5e19a7b67e8","Type":"ContainerStarted","Data":"9407ad695affd72a91ddb2f9375252d6aaec9c9d3947c19f6ae81ee25bcb05ed"} Jul 14 22:37:49.354923 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:49.354900 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc" Jul 14 22:37:49.480683 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:49.480652 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/081cd25d-f9ef-4900-bf3a-e5e19a7b67e8-maas-api-service-ca\") pod \"081cd25d-f9ef-4900-bf3a-e5e19a7b67e8\" (UID: \"081cd25d-f9ef-4900-bf3a-e5e19a7b67e8\") " Jul 14 22:37:49.481062 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:49.481004 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081cd25d-f9ef-4900-bf3a-e5e19a7b67e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "081cd25d-f9ef-4900-bf3a-e5e19a7b67e8" (UID: "081cd25d-f9ef-4900-bf3a-e5e19a7b67e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:37:49.581701 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:49.581667 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/081cd25d-f9ef-4900-bf3a-e5e19a7b67e8-maas-api-service-ca\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:37:50.239164 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:50.239131 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc" Jul 14 22:37:50.239164 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:50.239150 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc" event={"ID":"081cd25d-f9ef-4900-bf3a-e5e19a7b67e8","Type":"ContainerDied","Data":"9407ad695affd72a91ddb2f9375252d6aaec9c9d3947c19f6ae81ee25bcb05ed"} Jul 14 22:37:50.239391 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:37:50.239205 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9407ad695affd72a91ddb2f9375252d6aaec9c9d3947c19f6ae81ee25bcb05ed" Jul 14 22:38:12.697708 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:12.697674 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p"] Jul 14 22:38:12.698260 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:12.697940 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" containerID="cri-o://daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411" gracePeriod=30 Jul 14 22:38:17.947060 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:17.947009 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:38:18.037594 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037564 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-envoy\") pod \"a786c041-5700-4310-9c47-965c43a3e7dd\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " Jul 14 22:38:18.037594 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037597 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-credential-socket\") pod \"a786c041-5700-4310-9c47-965c43a3e7dd\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " Jul 14 22:38:18.037857 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037633 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a786c041-5700-4310-9c47-965c43a3e7dd-istio-podinfo\") pod \"a786c041-5700-4310-9c47-965c43a3e7dd\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " Jul 14 22:38:18.037857 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037656 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-istio-token\") pod \"a786c041-5700-4310-9c47-965c43a3e7dd\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " Jul 14 22:38:18.037857 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037686 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-socket\") pod \"a786c041-5700-4310-9c47-965c43a3e7dd\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " Jul 14 22:38:18.037857 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037726 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-data\") pod \"a786c041-5700-4310-9c47-965c43a3e7dd\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " Jul 14 22:38:18.037857 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037746 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-certs\") pod \"a786c041-5700-4310-9c47-965c43a3e7dd\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " Jul 14 22:38:18.037857 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037772 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qlcl\" (UniqueName: \"kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-kube-api-access-6qlcl\") pod \"a786c041-5700-4310-9c47-965c43a3e7dd\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " Jul 14 22:38:18.037857 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037816 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a786c041-5700-4310-9c47-965c43a3e7dd-istiod-ca-cert\") pod \"a786c041-5700-4310-9c47-965c43a3e7dd\" (UID: \"a786c041-5700-4310-9c47-965c43a3e7dd\") " Jul 14 22:38:18.038363 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037909 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a786c041-5700-4310-9c47-965c43a3e7dd" (UID: "a786c041-5700-4310-9c47-965c43a3e7dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:18.038363 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.037991 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a786c041-5700-4310-9c47-965c43a3e7dd" (UID: "a786c041-5700-4310-9c47-965c43a3e7dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:18.038363 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.038012 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a786c041-5700-4310-9c47-965c43a3e7dd" (UID: "a786c041-5700-4310-9c47-965c43a3e7dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:18.038363 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.038255 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:18.038363 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.038275 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:18.038363 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.038291 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:18.038363 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.038329 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a786c041-5700-4310-9c47-965c43a3e7dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a786c041-5700-4310-9c47-965c43a3e7dd" (UID: "a786c041-5700-4310-9c47-965c43a3e7dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:38:18.038779 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.038748 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "a786c041-5700-4310-9c47-965c43a3e7dd" (UID: "a786c041-5700-4310-9c47-965c43a3e7dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:18.039983 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.039954 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a786c041-5700-4310-9c47-965c43a3e7dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a786c041-5700-4310-9c47-965c43a3e7dd" (UID: "a786c041-5700-4310-9c47-965c43a3e7dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:38:18.039983 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.039967 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-kube-api-access-6qlcl" (OuterVolumeSpecName: "kube-api-access-6qlcl") pod "a786c041-5700-4310-9c47-965c43a3e7dd" (UID: "a786c041-5700-4310-9c47-965c43a3e7dd"). InnerVolumeSpecName "kube-api-access-6qlcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:38:18.040156 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.040011 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a786c041-5700-4310-9c47-965c43a3e7dd" (UID: "a786c041-5700-4310-9c47-965c43a3e7dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:18.040156 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.040013 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "a786c041-5700-4310-9c47-965c43a3e7dd" (UID: "a786c041-5700-4310-9c47-965c43a3e7dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:38:18.138792 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.138752 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:18.138792 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.138784 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qlcl\" (UniqueName: \"kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-kube-api-access-6qlcl\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:18.138792 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.138797 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a786c041-5700-4310-9c47-965c43a3e7dd-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:18.139071 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.138809 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a786c041-5700-4310-9c47-965c43a3e7dd-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:18.139071 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.138821 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a786c041-5700-4310-9c47-965c43a3e7dd-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:18.139071 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.138831 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a786c041-5700-4310-9c47-965c43a3e7dd-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:18.400322 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.400233 2562 generic.go:356] "Generic (PLEG): container finished" podID="a786c041-5700-4310-9c47-965c43a3e7dd" containerID="daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411" exitCode=0 Jul 14 22:38:18.400322 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.400279 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" Jul 14 22:38:18.400517 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.400279 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" event={"ID":"a786c041-5700-4310-9c47-965c43a3e7dd","Type":"ContainerDied","Data":"daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411"} Jul 14 22:38:18.400517 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.400410 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p" event={"ID":"a786c041-5700-4310-9c47-965c43a3e7dd","Type":"ContainerDied","Data":"8de9607d31d89abffba683bab62d02447886aad0315b902802a2a541affeba1b"} Jul 14 22:38:18.400517 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.400449 2562 scope.go:117] "RemoveContainer" containerID="daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411" Jul 14 22:38:18.410687 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.410669 2562 scope.go:117] "RemoveContainer" containerID="ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5" Jul 14 22:38:18.419813 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.419795 2562 scope.go:117] "RemoveContainer" containerID="daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411" Jul 14 22:38:18.420071 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:38:18.420051 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411\": container with ID starting with daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411 not found: ID does not exist" containerID="daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411" Jul 14 22:38:18.420156 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.420084 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411"} err="failed to get container status \"daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411\": rpc error: code = NotFound desc = could not find container \"daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411\": container with ID starting with daca2bf450b2fe3e87fee7d6e6b79b28580e922ea6961e68554c5ca0d22dc411 not found: ID does not exist" Jul 14 22:38:18.420156 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.420109 2562 scope.go:117] "RemoveContainer" containerID="ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5" Jul 14 22:38:18.420325 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:38:18.420308 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5\": container with ID starting with ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5 not found: ID does not exist" containerID="ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5" Jul 14 22:38:18.420379 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.420335 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5"} err="failed to get container status \"ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5\": rpc error: code = NotFound desc = could not find container \"ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5\": container with ID starting with ab98b675b604b1ab351c616030b81d877906cdaf8696d481f1fa48420dfca4d5 not found: ID does not exist" Jul 14 22:38:18.423823 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.423801 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p"] Jul 14 22:38:18.428515 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.428494 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-fdfd38-openshift-default-7688c6fdc5-xdf5p"] Jul 14 22:38:18.810828 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:18.810793 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" path="/var/lib/kubelet/pods/a786c041-5700-4310-9c47-965c43a3e7dd/volumes" Jul 14 22:38:38.848467 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:38.848428 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75"] Jul 14 22:38:38.848909 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:38.848628 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" containerID="cri-o://bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98" gracePeriod=30 Jul 14 22:38:44.104171 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.104143 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:38:44.281107 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281073 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-socket\") pod \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " Jul 14 22:38:44.281107 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281108 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istiod-ca-cert\") pod \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " Jul 14 22:38:44.281337 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281145 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-podinfo\") pod \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " Jul 14 22:38:44.281337 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281165 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f9pb\" (UniqueName: \"kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-kube-api-access-6f9pb\") pod \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " Jul 14 22:38:44.281337 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281186 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-token\") pod \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " Jul 14 22:38:44.281506 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281385 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42e24446-e3da-4f16-bb49-185ee5cd2ab9" (UID: "42e24446-e3da-4f16-bb49-185ee5cd2ab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:44.281506 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281397 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-envoy\") pod \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " Jul 14 22:38:44.281506 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281438 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-data\") pod \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " Jul 14 22:38:44.281506 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281487 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-credential-socket\") pod \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " Jul 14 22:38:44.281728 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281541 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-certs\") pod \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\" (UID: \"42e24446-e3da-4f16-bb49-185ee5cd2ab9\") " Jul 14 22:38:44.281728 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281609 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42e24446-e3da-4f16-bb49-185ee5cd2ab9" (UID: "42e24446-e3da-4f16-bb49-185ee5cd2ab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:38:44.281885 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281862 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42e24446-e3da-4f16-bb49-185ee5cd2ab9" (UID: "42e24446-e3da-4f16-bb49-185ee5cd2ab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:44.281949 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281890 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:44.281949 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281913 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istiod-ca-cert\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:44.282082 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.281957 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42e24446-e3da-4f16-bb49-185ee5cd2ab9" (UID: "42e24446-e3da-4f16-bb49-185ee5cd2ab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:44.282477 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.282442 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-data" (OuterVolumeSpecName: "istio-data") pod "42e24446-e3da-4f16-bb49-185ee5cd2ab9" (UID: "42e24446-e3da-4f16-bb49-185ee5cd2ab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:44.283545 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.283518 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-kube-api-access-6f9pb" (OuterVolumeSpecName: "kube-api-access-6f9pb") pod "42e24446-e3da-4f16-bb49-185ee5cd2ab9" (UID: "42e24446-e3da-4f16-bb49-185ee5cd2ab9"). InnerVolumeSpecName "kube-api-access-6f9pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:38:44.283545 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.283536 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-token" (OuterVolumeSpecName: "istio-token") pod "42e24446-e3da-4f16-bb49-185ee5cd2ab9" (UID: "42e24446-e3da-4f16-bb49-185ee5cd2ab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:38:44.283760 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.283735 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42e24446-e3da-4f16-bb49-185ee5cd2ab9" (UID: "42e24446-e3da-4f16-bb49-185ee5cd2ab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:44.284172 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.284148 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42e24446-e3da-4f16-bb49-185ee5cd2ab9" (UID: "42e24446-e3da-4f16-bb49-185ee5cd2ab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:38:44.382719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.382680 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-envoy\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:44.382719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.382711 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-data\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:44.382719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.382721 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-credential-socket\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:44.382719 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.382730 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e24446-e3da-4f16-bb49-185ee5cd2ab9-workload-certs\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:44.382992 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.382739 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-podinfo\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:44.382992 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.382748 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f9pb\" (UniqueName: \"kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-kube-api-access-6f9pb\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:44.382992 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.382757 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e24446-e3da-4f16-bb49-185ee5cd2ab9-istio-token\") on node \"ip-10-0-141-219.ec2.internal\" DevicePath \"\"" Jul 14 22:38:44.553195 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.553106 2562 generic.go:356] "Generic (PLEG): container finished" podID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerID="bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98" exitCode=0 Jul 14 22:38:44.553195 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.553159 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" Jul 14 22:38:44.553384 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.553195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" event={"ID":"42e24446-e3da-4f16-bb49-185ee5cd2ab9","Type":"ContainerDied","Data":"bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98"} Jul 14 22:38:44.553384 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.553257 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75" event={"ID":"42e24446-e3da-4f16-bb49-185ee5cd2ab9","Type":"ContainerDied","Data":"11d4d40911cfa603c5dca7a0ce4eced688ffe8e297556dffad0441b5e074a9ff"} Jul 14 22:38:44.553384 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.553288 2562 scope.go:117] "RemoveContainer" containerID="bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98" Jul 14 22:38:44.565854 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.565832 2562 scope.go:117] "RemoveContainer" containerID="e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209" Jul 14 22:38:44.574942 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.574920 2562 scope.go:117] "RemoveContainer" containerID="bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98" Jul 14 22:38:44.575287 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:38:44.575259 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98\": container with ID starting with bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98 not found: ID does not exist" containerID="bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98" Jul 14 22:38:44.575367 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.575302 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98"} err="failed to get container status \"bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98\": rpc error: code = NotFound desc = could not find container \"bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98\": container with ID starting with bdc8185a629ce0a8e744aa3d994dae3fe3fdea7b5130d71382ca26bb8aebfe98 not found: ID does not exist" Jul 14 22:38:44.575367 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.575329 2562 scope.go:117] "RemoveContainer" containerID="e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209" Jul 14 22:38:44.575655 ip-10-0-141-219 kubenswrapper[2562]: E0714 22:38:44.575602 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209\": container with ID starting with e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209 not found: ID does not exist" containerID="e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209" Jul 14 22:38:44.575716 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.575665 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209"} err="failed to get container status \"e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209\": rpc error: code = NotFound desc = could not find container \"e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209\": container with ID starting with e4bce8a4562c42c430f38c8019feb4d2bdeab25e8aae65023c8123362cfc6209 not found: ID does not exist" Jul 14 22:38:44.577962 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.577941 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75"] Jul 14 22:38:44.581392 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.581368 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-32d7b3-openshift-default-6b85d98bf9-lmg75"] Jul 14 22:38:44.810388 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:44.810301 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" path="/var/lib/kubelet/pods/42e24446-e3da-4f16-bb49-185ee5cd2ab9/volumes" Jul 14 22:38:45.157039 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:45.156948 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c99649887-rgv27_257db741-246e-4426-818a-54585d5ddb7b/copy-manifests/0.log" Jul 14 22:38:45.170373 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:45.170346 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c99649887-rgv27_257db741-246e-4426-818a-54585d5ddb7b/manager/0.log" Jul 14 22:38:45.690802 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:45.690771 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734455-cm55z_c42c7458-2bb7-4586-8ee4-ef05755b7db9/cleanup/0.log" Jul 14 22:38:45.930171 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:45.930141 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-5574436c-g5gtn_6a3d8ee8-7ee9-4ea1-a04f-d18a61110353/revoke-keys/0.log" Jul 14 22:38:46.053839 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:46.053816 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-c29be32a-p82sm_4f988bbd-539e-43c1-af4b-2252ea15c184/revoke-keys/0.log" Jul 14 22:38:46.164472 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:46.164436 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-de6a00-gjqrr_e9e87752-6ad2-4410-bf7f-2dab6345c696/revoke-keys/0.log" Jul 14 22:38:46.380375 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:46.380290 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-6b2eb522-qx2tp_f085c840-e51a-4239-be21-6c7c80e0ac84/revoke-keys/0.log" Jul 14 22:38:46.494489 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:46.494460 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-6a0cfb-mmqkv_c5c5b0e2-b877-4ddc-8174-0e3a9180c7cf/revoke-keys/0.log" Jul 14 22:38:46.709967 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:46.709889 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-de92a977-x4rnb_6c973853-901f-47cd-a000-1159937992e3/revoke-keys/0.log" Jul 14 22:38:46.828384 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:46.828350 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-ba078dbe-72snc_53a43c3e-1703-48a2-9bb5-7cd8896e03da/revoke-keys/0.log" Jul 14 22:38:46.950669 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:46.950643 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-c488720f-5wmqk_3132de05-86ab-4fa8-8089-7143a0b62b09/revoke-keys/0.log" Jul 14 22:38:47.074610 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:47.074577 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-fdfd38-5pjtc_081cd25d-f9ef-4900-bf3a-e5e19a7b67e8/revoke-keys/0.log" Jul 14 22:38:48.078955 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.078931 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq_fadea2e5-5b30-49d6-974c-8675e4b39316/util/0.log" Jul 14 22:38:48.085418 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.085390 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq_fadea2e5-5b30-49d6-974c-8675e4b39316/pull/0.log" Jul 14 22:38:48.091959 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.091939 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq_fadea2e5-5b30-49d6-974c-8675e4b39316/extract/0.log" Jul 14 22:38:48.202491 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.202468 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj_766bf267-dc3d-46ba-8b15-d918114ead9d/util/0.log" Jul 14 22:38:48.208844 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.208815 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj_766bf267-dc3d-46ba-8b15-d918114ead9d/pull/0.log" Jul 14 22:38:48.214917 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.214895 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj_766bf267-dc3d-46ba-8b15-d918114ead9d/extract/0.log" Jul 14 22:38:48.321843 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.321816 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn_02eb97d0-9f26-4698-87c2-a96b81013779/util/0.log" Jul 14 22:38:48.327037 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.327001 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn_02eb97d0-9f26-4698-87c2-a96b81013779/pull/0.log" Jul 14 22:38:48.333379 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.333324 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn_02eb97d0-9f26-4698-87c2-a96b81013779/extract/0.log" Jul 14 22:38:48.445450 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.445419 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f_c30f78c7-01aa-4399-af9d-3e30a1da8411/util/0.log" Jul 14 22:38:48.452749 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.452725 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f_c30f78c7-01aa-4399-af9d-3e30a1da8411/pull/0.log" Jul 14 22:38:48.459403 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:48.459384 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f_c30f78c7-01aa-4399-af9d-3e30a1da8411/extract/0.log" Jul 14 22:38:49.749674 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:49.749641 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6787xc8n_bedecead-83e4-425c-bac2-4d58a627cf2a/istio-proxy/0.log" Jul 14 22:38:49.871383 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:49.871344 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wc68_6f7825e5-d4b5-489f-a4c2-32ab490c384c/discovery/0.log" Jul 14 22:38:50.230356 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:50.230326 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vqgb9_494a30d6-6cb0-4f37-9f22-753ac3f31fb5/istio-proxy/1.log" Jul 14 22:38:50.464534 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:50.464505 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-kfknj_1c8397de-5442-4164-8820-24b34ba09a0c/payload-processing/0.log" Jul 14 22:38:58.899571 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:58.899492 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8dxrp_c7f4d121-c802-43d7-995b-4a54e3d71205/global-pull-secret-syncer/0.log" Jul 14 22:38:59.017952 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:59.017922 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rw49t_a3383a4e-85df-440d-9ab7-ba09e045389a/konnectivity-agent/0.log" Jul 14 22:38:59.063487 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:38:59.063443 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-219.ec2.internal_355c9c11aac4261fc69af718f808c587/haproxy/0.log" Jul 14 22:39:04.167649 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.167617 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq_fadea2e5-5b30-49d6-974c-8675e4b39316/extract/0.log" Jul 14 22:39:04.185188 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.185163 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq_fadea2e5-5b30-49d6-974c-8675e4b39316/util/0.log" Jul 14 22:39:04.216358 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.216328 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dpdq_fadea2e5-5b30-49d6-974c-8675e4b39316/pull/0.log" Jul 14 22:39:04.246651 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.246624 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj_766bf267-dc3d-46ba-8b15-d918114ead9d/extract/0.log" Jul 14 22:39:04.270227 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.270200 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj_766bf267-dc3d-46ba-8b15-d918114ead9d/util/0.log" Jul 14 22:39:04.296408 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.296380 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02scmj_766bf267-dc3d-46ba-8b15-d918114ead9d/pull/0.log" Jul 14 22:39:04.329189 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.329166 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn_02eb97d0-9f26-4698-87c2-a96b81013779/extract/0.log" Jul 14 22:39:04.370623 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.370591 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn_02eb97d0-9f26-4698-87c2-a96b81013779/util/0.log" Jul 14 22:39:04.393325 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.393298 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739fxsn_02eb97d0-9f26-4698-87c2-a96b81013779/pull/0.log" Jul 14 22:39:04.425197 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.425111 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f_c30f78c7-01aa-4399-af9d-3e30a1da8411/extract/0.log" Jul 14 22:39:04.448679 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.448652 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f_c30f78c7-01aa-4399-af9d-3e30a1da8411/util/0.log" Jul 14 22:39:04.473339 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:04.473308 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m8k9f_c30f78c7-01aa-4399-af9d-3e30a1da8411/pull/0.log" Jul 14 22:39:06.940073 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:06.940047 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b405286f-706e-4f3b-8484-5dfb9284d912/alertmanager/0.log" Jul 14 22:39:06.974924 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:06.974899 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b405286f-706e-4f3b-8484-5dfb9284d912/config-reloader/0.log" Jul 14 22:39:07.004649 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.004620 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b405286f-706e-4f3b-8484-5dfb9284d912/kube-rbac-proxy-web/0.log" Jul 14 22:39:07.030386 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.030358 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b405286f-706e-4f3b-8484-5dfb9284d912/kube-rbac-proxy/0.log" Jul 14 22:39:07.057924 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.057898 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b405286f-706e-4f3b-8484-5dfb9284d912/kube-rbac-proxy-metric/0.log" Jul 14 22:39:07.083565 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.083538 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b405286f-706e-4f3b-8484-5dfb9284d912/prom-label-proxy/0.log" Jul 14 22:39:07.111718 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.111680 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b405286f-706e-4f3b-8484-5dfb9284d912/init-config-reloader/0.log" Jul 14 22:39:07.187996 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.187965 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5f67q_104b706a-2784-45de-8889-3b9469d51c72/kube-state-metrics/0.log" Jul 14 22:39:07.209529 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.209503 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5f67q_104b706a-2784-45de-8889-3b9469d51c72/kube-rbac-proxy-main/0.log" Jul 14 22:39:07.249767 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.249736 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5f67q_104b706a-2784-45de-8889-3b9469d51c72/kube-rbac-proxy-self/0.log" Jul 14 22:39:07.321886 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.321858 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-b77hq_07efdcff-0877-4700-83cd-441042289f78/monitoring-plugin/0.log" Jul 14 22:39:07.362092 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.362065 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4xhpc_f6656125-2156-4760-9a62-a7713a16e946/node-exporter/0.log" Jul 14 22:39:07.388274 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.388246 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4xhpc_f6656125-2156-4760-9a62-a7713a16e946/kube-rbac-proxy/0.log" Jul 14 22:39:07.411765 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.411741 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4xhpc_f6656125-2156-4760-9a62-a7713a16e946/init-textfile/0.log" Jul 14 22:39:07.895574 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.895495 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-mnwlg_c620f4d1-b9ca-456e-abd3-a6982435ca5e/prometheus-operator/0.log" Jul 14 22:39:07.916827 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.916802 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-mnwlg_c620f4d1-b9ca-456e-abd3-a6982435ca5e/kube-rbac-proxy/0.log" Jul 14 22:39:07.947687 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:07.947663 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-s8wbl_823b44cc-8073-4cd0-9698-bf2d1975fccb/prometheus-operator-admission-webhook/0.log" Jul 14 22:39:09.211680 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.211629 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm"] Jul 14 22:39:09.212247 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212227 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" Jul 14 22:39:09.212328 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212250 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" Jul 14 22:39:09.212328 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212275 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" Jul 14 22:39:09.212328 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212283 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" Jul 14 22:39:09.212328 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212299 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" Jul 14 22:39:09.212328 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212307 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" Jul 14 22:39:09.212328 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212318 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="081cd25d-f9ef-4900-bf3a-e5e19a7b67e8" containerName="revoke-keys" Jul 14 22:39:09.212328 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212328 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="081cd25d-f9ef-4900-bf3a-e5e19a7b67e8" containerName="revoke-keys" Jul 14 22:39:09.212691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212345 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" Jul 14 22:39:09.212691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212354 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" Jul 14 22:39:09.212691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212454 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" Jul 14 22:39:09.212691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212468 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" Jul 14 22:39:09.212691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212480 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a786c041-5700-4310-9c47-965c43a3e7dd" containerName="istio-proxy" Jul 14 22:39:09.212691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212490 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e24446-e3da-4f16-bb49-185ee5cd2ab9" containerName="istio-proxy" Jul 14 22:39:09.212691 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.212500 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="081cd25d-f9ef-4900-bf3a-e5e19a7b67e8" containerName="revoke-keys" Jul 14 22:39:09.217242 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.217221 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.219575 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.219552 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tpwxp\"/\"kube-root-ca.crt\"" Jul 14 22:39:09.220558 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.220538 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tpwxp\"/\"openshift-service-ca.crt\"" Jul 14 22:39:09.220558 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.220553 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tpwxp\"/\"default-dockercfg-m2k6m\"" Jul 14 22:39:09.225280 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.225258 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm"] Jul 14 22:39:09.292443 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.292399 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-lib-modules\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.292633 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.292455 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-proc\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.292633 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.292533 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-sys\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.292633 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.292578 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbrzp\" (UniqueName: \"kubernetes.io/projected/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-kube-api-access-lbrzp\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.292633 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.292614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-podres\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.393852 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.393812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-sys\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.394052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.393862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbrzp\" (UniqueName: \"kubernetes.io/projected/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-kube-api-access-lbrzp\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.394052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.393906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-podres\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.394052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.393948 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-sys\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.394052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.393967 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-lib-modules\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.394052 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.393999 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-proc\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.394237 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.394096 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-podres\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.394237 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.394103 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-proc\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.394237 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.394147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-lib-modules\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.401720 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.401688 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbrzp\" (UniqueName: \"kubernetes.io/projected/c9546a7f-9278-4f73-a183-8c7cbf3a0a18-kube-api-access-lbrzp\") pod \"perf-node-gather-daemonset-zqpwm\" (UID: \"c9546a7f-9278-4f73-a183-8c7cbf3a0a18\") " pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.408223 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.408199 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-f4fvf_4481f524-9474-4352-baf0-994335283f40/networking-console-plugin/0.log" Jul 14 22:39:09.528609 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.528579 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:09.665152 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.665118 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm"] Jul 14 22:39:09.666434 ip-10-0-141-219 kubenswrapper[2562]: W0714 22:39:09.666406 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc9546a7f_9278_4f73_a183_8c7cbf3a0a18.slice/crio-5608c5fe0fc6a043030d53070ba02d3175b586e2cad592909a0a8ca22e47f791 WatchSource:0}: Error finding container 5608c5fe0fc6a043030d53070ba02d3175b586e2cad592909a0a8ca22e47f791: Status 404 returned error can't find the container with id 5608c5fe0fc6a043030d53070ba02d3175b586e2cad592909a0a8ca22e47f791 Jul 14 22:39:09.701912 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:09.701880 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" event={"ID":"c9546a7f-9278-4f73-a183-8c7cbf3a0a18","Type":"ContainerStarted","Data":"5608c5fe0fc6a043030d53070ba02d3175b586e2cad592909a0a8ca22e47f791"} Jul 14 22:39:10.549835 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:10.549808 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-d8744fcb7-fr6gx_4cfa8389-30cd-4bc5-aaa5-65acd9e75b2b/console/0.log" Jul 14 22:39:10.587037 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:10.586996 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-jf8ff_fa555683-9f61-43a9-a9a6-039274be0052/download-server/0.log" Jul 14 22:39:10.708593 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:10.708555 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" event={"ID":"c9546a7f-9278-4f73-a183-8c7cbf3a0a18","Type":"ContainerStarted","Data":"52775b61091a7647d37d162d9251ccc4e1d7d66dae82925e49aafd3c70da707e"} Jul 14 22:39:10.708809 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:10.708632 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:10.727951 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:10.727898 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" podStartSLOduration=1.727884024 podStartE2EDuration="1.727884024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:39:10.724937268 +0000 UTC m=+2885.574407333" watchObservedRunningTime="2026-07-14 22:39:10.727884024 +0000 UTC m=+2885.577354064" Jul 14 22:39:11.187349 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:11.187320 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-q6q9z_7e0ae623-c7d3-480b-9f2f-dfe60a436ad2/volume-data-source-validator/0.log" Jul 14 22:39:12.283109 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:12.283073 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2krhj_3380677a-ab5c-4196-864f-891f7cb4a5ed/dns/0.log" Jul 14 22:39:12.309690 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:12.309663 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2krhj_3380677a-ab5c-4196-864f-891f7cb4a5ed/kube-rbac-proxy/0.log" Jul 14 22:39:12.452129 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:12.452099 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b4ttn_6212dead-9e15-43dc-bbef-57928fbd39f1/dns-node-resolver/0.log" Jul 14 22:39:13.157354 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:13.157329 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-frswj_38b6f451-563f-498b-86d9-18dbd2f78850/node-ca/0.log" Jul 14 22:39:14.187710 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:14.187682 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6787xc8n_bedecead-83e4-425c-bac2-4d58a627cf2a/istio-proxy/0.log" Jul 14 22:39:14.313931 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:14.313901 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wc68_6f7825e5-d4b5-489f-a4c2-32ab490c384c/discovery/0.log" Jul 14 22:39:14.395253 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:14.395195 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vqgb9_494a30d6-6cb0-4f37-9f22-753ac3f31fb5/istio-proxy/0.log" Jul 14 22:39:14.429369 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:14.429339 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vqgb9_494a30d6-6cb0-4f37-9f22-753ac3f31fb5/istio-proxy/1.log" Jul 14 22:39:14.493660 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:14.493578 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-kfknj_1c8397de-5442-4164-8820-24b34ba09a0c/payload-processing/0.log" Jul 14 22:39:15.237580 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:15.237550 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bnc47_764d3fb0-f902-4449-b1c7-ae0abdaa8a6d/serve-healthcheck-canary/0.log" Jul 14 22:39:15.927301 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:15.927242 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pkp6j_5fbfba53-2715-4194-9075-cf7ff451053e/kube-rbac-proxy/0.log" Jul 14 22:39:15.947327 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:15.947299 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pkp6j_5fbfba53-2715-4194-9075-cf7ff451053e/exporter/0.log" Jul 14 22:39:15.969695 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:15.969650 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pkp6j_5fbfba53-2715-4194-9075-cf7ff451053e/extractor/0.log" Jul 14 22:39:16.723750 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:16.723720 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tpwxp/perf-node-gather-daemonset-zqpwm" Jul 14 22:39:18.945997 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:18.945911 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c99649887-rgv27_257db741-246e-4426-818a-54585d5ddb7b/manager/0.log" Jul 14 22:39:18.971804 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:18.971778 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c99649887-rgv27_257db741-246e-4426-818a-54585d5ddb7b/copy-manifests/0.log" Jul 14 22:39:20.303602 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:20.303578 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-676d99795d-d8gv9_8728d6cf-3089-45f6-b92a-c716e967ade4/manager/0.log" Jul 14 22:39:20.360367 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:20.360335 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6787xc8n_bedecead-83e4-425c-bac2-4d58a627cf2a/istio-proxy/0.log" Jul 14 22:39:20.479935 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:20.479904 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wc68_6f7825e5-d4b5-489f-a4c2-32ab490c384c/discovery/0.log" Jul 14 22:39:20.533778 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:20.533740 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vqgb9_494a30d6-6cb0-4f37-9f22-753ac3f31fb5/istio-proxy/0.log" Jul 14 22:39:20.562774 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:20.562700 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vqgb9_494a30d6-6cb0-4f37-9f22-753ac3f31fb5/istio-proxy/1.log" Jul 14 22:39:20.612955 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:20.612922 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-kfknj_1c8397de-5442-4164-8820-24b34ba09a0c/payload-processing/0.log" Jul 14 22:39:21.899623 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:21.899591 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6787xc8n_bedecead-83e4-425c-bac2-4d58a627cf2a/istio-proxy/0.log" Jul 14 22:39:22.010390 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:22.010357 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wc68_6f7825e5-d4b5-489f-a4c2-32ab490c384c/discovery/0.log" Jul 14 22:39:22.065428 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:22.065398 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vqgb9_494a30d6-6cb0-4f37-9f22-753ac3f31fb5/istio-proxy/0.log" Jul 14 22:39:22.096298 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:22.096269 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vqgb9_494a30d6-6cb0-4f37-9f22-753ac3f31fb5/istio-proxy/1.log" Jul 14 22:39:22.141730 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:22.141702 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-kfknj_1c8397de-5442-4164-8820-24b34ba09a0c/payload-processing/0.log" Jul 14 22:39:25.506746 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:25.506716 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-f4glr_ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5/migrator/0.log" Jul 14 22:39:25.528641 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:25.528613 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-f4glr_ac0d44ba-0ebd-46be-bfd8-9278fc18b0b5/graceful-termination/0.log" Jul 14 22:39:27.106163 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:27.106132 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cf2lf_535d8ec6-5b58-40eb-b09f-b8b3a8044071/kube-multus-additional-cni-plugins/0.log" Jul 14 22:39:27.129585 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:27.129557 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cf2lf_535d8ec6-5b58-40eb-b09f-b8b3a8044071/egress-router-binary-copy/0.log" Jul 14 22:39:27.153766 ip-10-0-141-219 kubenswrapper[2562]: I0714 22:39:27.153736 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cf2lf_535d8ec6-5b58-40eb-b09f-b8b3a8044071/cni-plugins/0.log"