Jul 07 13:27:44.758598 ip-10-0-132-236 systemd[1]: Starting Kubernetes Kubelet... Jul 07 13:27:46.310330 ip-10-0-132-236 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 13:27:46.310330 ip-10-0-132-236 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 13:27:46.310330 ip-10-0-132-236 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 13:27:46.310330 ip-10-0-132-236 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 13:27:46.310330 ip-10-0-132-236 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 13:27:46.310976 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.310315 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 13:27:46.315349 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315333 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 13:27:46.315349 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315348 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315352 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315355 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315359 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315362 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315365 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315367 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315371 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315374 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315377 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315380 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315383 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315385 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315388 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315391 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315393 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315396 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315403 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315406 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315409 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 13:27:46.315417 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315411 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315414 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315416 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315419 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315423 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315428 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315431 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315434 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315436 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315439 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315442 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315445 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315447 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315450 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315453 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315455 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315458 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315461 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315465 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 13:27:46.315908 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315468 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315471 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315474 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315478 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315482 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315485 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315488 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315491 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315494 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315497 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315499 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315502 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315504 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315507 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315509 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315513 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315516 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315518 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315521 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 13:27:46.316394 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315524 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315527 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315530 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315532 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315535 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315537 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315541 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315544 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315547 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315549 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315552 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315555 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315557 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315560 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315562 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315565 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315568 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315570 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315573 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 13:27:46.316885 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315576 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315578 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315581 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315583 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315586 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315589 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315591 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315594 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315596 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315599 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315602 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315605 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315607 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315610 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315612 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315615 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315617 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315620 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315622 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315625 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 13:27:46.317353 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.315628 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316338 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316345 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316349 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316352 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316354 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316357 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316360 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316362 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316365 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316367 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316370 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316372 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316375 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316378 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316380 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316383 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316386 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316388 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316391 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 13:27:46.317822 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316393 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316396 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316399 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316402 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316405 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316407 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316410 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316413 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316415 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316418 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316420 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316423 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316425 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316428 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316430 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316433 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316436 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316439 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316444 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 13:27:46.318318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316448 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316452 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316455 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316458 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316461 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316464 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316467 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316470 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316473 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316475 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316479 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316483 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316485 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316488 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316490 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316493 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316495 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316498 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316500 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316503 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 13:27:46.318777 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316505 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316508 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316510 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316513 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316515 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316517 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316520 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316522 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316525 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316527 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316530 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316532 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316535 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316537 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316540 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316542 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316545 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316547 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316550 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316552 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 13:27:46.319260 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316555 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316558 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316563 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316566 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316569 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316571 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316575 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316579 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316582 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316585 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316588 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316591 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316593 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316596 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316598 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316601 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316604 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316606 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316608 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 13:27:46.319739 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316611 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316613 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316989 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316994 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.316997 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317000 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317003 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317005 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317008 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317011 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317013 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317016 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317019 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317022 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317026 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317031 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317035 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317038 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317041 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317044 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 13:27:46.320220 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317046 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317049 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317051 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317054 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317057 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317059 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317062 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317064 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317067 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317069 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317072 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317074 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317077 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317079 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317082 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317084 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317087 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317090 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317092 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317094 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 13:27:46.320724 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317098 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317100 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317102 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317105 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317108 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317111 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317113 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317119 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317122 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317125 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317127 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317132 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317135 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317138 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317141 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317143 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317146 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317149 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317152 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 13:27:46.321224 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317156 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317159 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317176 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317179 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317182 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317184 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317187 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317189 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317192 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317195 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317197 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317200 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317202 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317205 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317207 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317210 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317213 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317215 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317218 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317220 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 13:27:46.321669 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317224 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317227 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317230 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317233 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317236 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317238 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317241 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317244 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317247 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317250 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317252 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317255 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317258 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317260 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317262 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317265 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317267 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317270 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317272 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317275 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 13:27:46.322198 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317278 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317280 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317346 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317353 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317359 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317364 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317368 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317371 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317375 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317380 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317383 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317386 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317391 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317394 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317398 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317401 2576 flags.go:64] FLAG: --cgroup-root="" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317404 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317407 2576 flags.go:64] FLAG: --client-ca-file="" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317411 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317413 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317417 2576 flags.go:64] FLAG: --cluster-domain="" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317419 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317423 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317426 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 13:27:46.322682 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317429 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317433 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317436 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317439 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317442 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317445 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317448 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317451 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317456 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317458 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317462 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317465 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317468 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317471 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317474 2576 flags.go:64] FLAG: --enable-server="true" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317478 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317482 2576 flags.go:64] FLAG: --event-burst="100" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317485 2576 flags.go:64] FLAG: --event-qps="50" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317488 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317490 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317495 2576 flags.go:64] FLAG: --eviction-hard="" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317499 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317502 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317505 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317508 2576 flags.go:64] FLAG: --eviction-soft="" Jul 07 13:27:46.323306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317512 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317514 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317517 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317520 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317523 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317526 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317528 2576 flags.go:64] FLAG: --feature-gates="" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317532 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317535 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317538 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317541 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317544 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317547 2576 flags.go:64] FLAG: --help="false" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317550 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317553 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317557 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317560 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317563 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317566 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317569 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317572 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317574 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317577 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 13:27:46.323886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317580 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317583 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317586 2576 flags.go:64] FLAG: --kube-reserved="" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317589 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317593 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317596 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317598 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317601 2576 flags.go:64] FLAG: --lock-file="" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317604 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317607 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317611 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317616 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317619 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317621 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317624 2576 flags.go:64] FLAG: --logging-format="text" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317627 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317630 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317633 2576 flags.go:64] FLAG: --manifest-url="" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317635 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317640 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317643 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317647 2576 flags.go:64] FLAG: --max-pods="110" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317650 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317653 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317657 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 13:27:46.324536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317660 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317663 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317666 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317669 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317676 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317679 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317682 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317685 2576 flags.go:64] FLAG: --pod-cidr="" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317688 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317693 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317696 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317699 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317703 2576 flags.go:64] FLAG: --port="10250" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317706 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317709 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-065541654f710688c" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317712 2576 flags.go:64] FLAG: --qos-reserved="" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317715 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317718 2576 flags.go:64] FLAG: --register-node="true" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317722 2576 flags.go:64] FLAG: --register-with-taints="" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317725 2576 flags.go:64] FLAG: --registry-burst="10" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317728 2576 flags.go:64] FLAG: --registry-qps="5" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317731 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317734 2576 flags.go:64] FLAG: --reserved-memory="" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317737 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317740 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 13:27:46.325565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317743 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317746 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317749 2576 flags.go:64] FLAG: --runonce="false" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317752 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317755 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317758 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317762 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317765 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317768 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317771 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317774 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317777 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317779 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317782 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317785 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317788 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317790 2576 flags.go:64] FLAG: --system-cgroups="" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317793 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317798 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317807 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317809 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317813 2576 flags.go:64] FLAG: --tls-min-version="" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317816 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317818 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317821 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 13:27:46.326580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317824 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317828 2576 flags.go:64] FLAG: --v="2" Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317832 2576 flags.go:64] FLAG: --version="false" Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317836 2576 flags.go:64] FLAG: --vmodule="" Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317840 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.317844 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317978 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317982 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317985 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317988 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317991 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317994 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.317996 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318000 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318003 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318005 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318008 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318010 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318013 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318015 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318018 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 13:27:46.327205 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318020 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318023 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318025 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318028 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318030 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318033 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318036 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318038 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318041 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318043 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318046 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318049 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318052 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318054 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318057 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318059 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318062 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318064 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318067 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318069 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 13:27:46.327748 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318071 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318074 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318076 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318079 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318083 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318085 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318087 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318090 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318093 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318095 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318097 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318100 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318102 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318105 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318107 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318109 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318112 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318115 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318118 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318120 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 13:27:46.328240 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318122 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318125 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318127 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318130 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318132 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318135 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318137 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318140 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318142 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318145 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318148 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318152 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318155 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318158 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318175 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318178 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318183 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318187 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318190 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 13:27:46.328732 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318193 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318196 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318199 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318202 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318205 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318208 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318211 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318215 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318217 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318220 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318223 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318226 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318228 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318231 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318234 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318236 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318239 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318241 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318243 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 13:27:46.329202 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318246 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318249 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318252 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318254 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318257 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.318259 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.318950 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.327411 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.327439 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.327458 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.327468 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327523 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327528 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327531 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327534 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 13:27:46.329659 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327537 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327540 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327542 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327545 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327547 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327550 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327553 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327555 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327558 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327561 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327563 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327566 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327569 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327572 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327575 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327578 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327580 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327584 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327588 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 13:27:46.330031 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327591 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327594 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327597 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327599 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327602 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327604 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327608 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327611 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327613 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327616 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327618 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327621 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327624 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327626 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327629 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327631 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327634 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327636 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327639 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327641 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 13:27:46.330584 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327644 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327646 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327649 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327652 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327655 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327657 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327660 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327664 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327668 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327671 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327674 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327677 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327680 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327683 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327685 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327688 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327691 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327693 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327697 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 13:27:46.331070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327700 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327703 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327706 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327708 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327711 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327713 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327716 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327718 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327721 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327723 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327726 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327728 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327731 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327734 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327736 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327739 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327741 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327744 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327747 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 13:27:46.331539 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327751 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327755 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327758 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327761 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327765 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327767 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327771 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327774 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327777 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327779 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327782 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327785 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327788 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327791 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327793 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327796 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327799 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 13:27:46.331975 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327801 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.327806 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327932 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327937 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327940 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327943 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327946 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327949 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327952 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327954 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327957 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327960 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327962 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327965 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327968 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 13:27:46.332432 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327971 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327973 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327976 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327980 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327983 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327987 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327989 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327992 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327995 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.327998 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328000 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328004 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328006 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328009 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328011 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328014 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328016 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328019 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328021 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 13:27:46.332785 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328024 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328026 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328029 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328031 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328034 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328036 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328038 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328041 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328044 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328046 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328049 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328051 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328054 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328057 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328060 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328062 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328065 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328067 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328069 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 13:27:46.333245 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328073 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328076 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328079 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328082 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328086 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328089 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328092 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328094 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328097 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328099 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328102 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328105 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328107 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328110 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328112 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328115 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328117 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328120 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328122 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328125 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 13:27:46.333804 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328127 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328130 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328132 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328135 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328137 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328140 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328142 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328145 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328148 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328150 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328153 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328155 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328158 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328160 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328180 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328187 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328190 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328193 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328197 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328201 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 13:27:46.334338 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328204 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328207 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328209 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328212 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328215 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328217 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328220 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:46.328222 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.328227 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.328328 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.330999 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.332482 2576 server.go:1019] "Starting client certificate rotation" Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.332589 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 13:27:46.334824 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.332626 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 13:27:46.335139 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.335048 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 13:27:46.338727 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.338710 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 13:27:46.346713 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.346698 2576 log.go:25] "Validated CRI v1 runtime API" Jul 07 13:27:46.350254 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.350239 2576 log.go:25] "Validated CRI v1 image API" Jul 07 13:27:46.351488 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.351472 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 13:27:46.353964 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.353937 2576 fs.go:135] Filesystem UUIDs: map[296b498b-e241-47cf-95c5-65de27537562:/dev/nvme0n1p4 48ba8560-9e65-4968-a7f0-a9863d3daa94:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 07 13:27:46.354008 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.353964 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 13:27:46.359269 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.359143 2576 manager.go:216] Machine: {Timestamp:2026-07-07 13:27:46.358433982 +0000 UTC m=+0.359675505 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3061362 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2822195fe10d621313067161214274 SystemUUID:ec282219-5fe1-0d62-1313-067161214274 BootID:2f1bd2b6-3ed5-4e37-88af-42047268bb0d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5b:00:29:31:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5b:00:29:31:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:85:aa:91:3a:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 13:27:46.359269 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.359256 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 13:27:46.359382 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.359327 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 13:27:46.362545 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.362519 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 13:27:46.362683 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.362551 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 13:27:46.363267 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.363257 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 13:27:46.363304 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.363270 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 13:27:46.363304 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.363279 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 13:27:46.363797 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.363788 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 13:27:46.364991 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.364975 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 13:27:46.365589 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.365579 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 13:27:46.369255 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.369244 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 07 13:27:46.369396 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.369386 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 13:27:46.369433 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.369416 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 07 13:27:46.369433 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.369425 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 13:27:46.369497 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.369439 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 13:27:46.369497 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.369448 2576 kubelet.go:398] "Adding apiserver pod source" Jul 07 13:27:46.369497 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.369457 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 13:27:46.369788 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.369776 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 13:27:46.369825 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.369796 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 13:27:46.371909 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.371892 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 13:27:46.374204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.374190 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 13:27:46.374252 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.374221 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 13:27:46.375381 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375359 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 13:27:46.375425 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375396 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 13:27:46.375425 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375408 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 13:27:46.375425 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375417 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 13:27:46.375516 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375431 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 13:27:46.375516 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375439 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 13:27:46.375516 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375451 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 13:27:46.375516 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375459 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 13:27:46.375516 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375468 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 13:27:46.375516 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375477 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 13:27:46.375516 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375489 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 13:27:46.375516 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.375499 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 13:27:46.377722 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.377710 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 13:27:46.377756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.377724 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 13:27:46.380808 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.380794 2576 server.go:1268] "Started kubelet" Jul 07 13:27:46.380997 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.380926 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 13:27:46.381065 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.381017 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 13:27:46.381521 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.381471 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 13:27:46.381595 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.381521 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 13:27:46.381595 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.381527 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 13:27:46.381595 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.381552 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 13:27:46.381724 ip-10-0-132-236 systemd[1]: Started Kubernetes Kubelet. Jul 07 13:27:46.383405 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.383392 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 07 13:27:46.385637 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.385619 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 13:27:46.387528 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.387509 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 13:27:46.388404 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.387403 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-236.ec2.internal.18c004ec8817c44d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-236.ec2.internal,UID:ip-10-0-132-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-236.ec2.internal,},FirstTimestamp:2026-07-07 13:27:46.380768333 +0000 UTC m=+0.382009851,LastTimestamp:2026-07-07 13:27:46.380768333 +0000 UTC m=+0.382009851,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-236.ec2.internal,}" Jul 07 13:27:46.392313 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.392294 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 13:27:46.392313 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.392307 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 13:27:46.392453 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.392437 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:46.393015 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.392992 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 13:27:46.393015 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393006 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 13:27:46.393143 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393023 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 13:27:46.393143 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393080 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 07 13:27:46.393143 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393088 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 07 13:27:46.393259 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393199 2576 factory.go:55] Registering systemd factory Jul 07 13:27:46.393259 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393214 2576 factory.go:223] Registration of the systemd container factory successfully Jul 07 13:27:46.393446 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393433 2576 factory.go:153] Registering CRI-O factory Jul 07 13:27:46.393514 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393450 2576 factory.go:223] Registration of the crio container factory successfully Jul 07 13:27:46.393514 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393496 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 13:27:46.393600 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393539 2576 factory.go:103] Registering Raw factory Jul 07 13:27:46.393600 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.393552 2576 manager.go:1196] Started watching for new ooms in manager Jul 07 13:27:46.394037 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.394024 2576 manager.go:349] Starting recovery of all containers Jul 07 13:27:46.402733 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.402682 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 13:27:46.402733 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.402702 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 13:27:46.403727 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.403705 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qhflg" Jul 07 13:27:46.406385 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.406370 2576 manager.go:354] Recovery completed Jul 07 13:27:46.410156 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.410140 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qhflg" Jul 07 13:27:46.410323 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.410311 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 13:27:46.412035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.412021 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientMemory" Jul 07 13:27:46.412092 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.412050 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 13:27:46.412092 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.412061 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientPID" Jul 07 13:27:46.413435 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.413415 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 13:27:46.413435 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.413428 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 13:27:46.413552 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.413443 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 07 13:27:46.413552 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.413465 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 13:27:46.414327 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.414312 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientMemory" Jul 07 13:27:46.414388 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.414337 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 13:27:46.414388 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.414348 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientPID" Jul 07 13:27:46.414585 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.414570 2576 policy_none.go:49] "None policy: Start" Jul 07 13:27:46.414585 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.414585 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 13:27:46.414704 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.414596 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 13:27:46.415848 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.415838 2576 policy_none.go:47] "Start" Jul 07 13:27:46.456770 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.456754 2576 manager.go:337] "Starting Device Plugin manager" Jul 07 13:27:46.456864 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.456791 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 13:27:46.456864 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.456800 2576 server.go:87] "Starting device plugin registration server" Jul 07 13:27:46.457026 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.457014 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 13:27:46.457071 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.457026 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 13:27:46.457840 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.457804 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 13:27:46.457986 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.457974 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 13:27:46.457986 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.457986 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 13:27:46.458207 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.458186 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 13:27:46.458306 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.458231 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:46.557871 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.557846 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 13:27:46.559014 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.558784 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientMemory" Jul 07 13:27:46.559014 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.558812 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 13:27:46.559014 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.558825 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientPID" Jul 07 13:27:46.559014 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.558861 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.573072 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.573018 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.573072 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.573041 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-236.ec2.internal\": node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:46.582178 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.582150 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 13:27:46.583314 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.583296 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 13:27:46.583409 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.583337 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 13:27:46.583409 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.583375 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 13:27:46.583512 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.583421 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 13:27:46.585286 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.585266 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 13:27:46.600261 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.600244 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:46.683795 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.683722 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal"] Jul 07 13:27:46.683895 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.683836 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 13:27:46.685582 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.685567 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientMemory" Jul 07 13:27:46.685645 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.685597 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 13:27:46.685645 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.685606 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientPID" Jul 07 13:27:46.686888 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.686877 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 13:27:46.687040 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.687027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.687076 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.687070 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 13:27:46.687697 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.687684 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientMemory" Jul 07 13:27:46.687753 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.687703 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 13:27:46.687753 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.687712 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientPID" Jul 07 13:27:46.687753 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.687740 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientMemory" Jul 07 13:27:46.687851 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.687762 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 13:27:46.687851 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.687772 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientPID" Jul 07 13:27:46.688954 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.688938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.688999 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.688966 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 13:27:46.689688 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.689673 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientMemory" Jul 07 13:27:46.689744 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.689698 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 13:27:46.689744 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.689708 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeHasSufficientPID" Jul 07 13:27:46.693840 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.693815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/505b89dd4f29cd1d6366a8221bca889f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal\" (UID: \"505b89dd4f29cd1d6366a8221bca889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.693899 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.693848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18bcd73158ca1a5b1c82fc4d26d8e309-config\") pod \"kube-apiserver-proxy-ip-10-0-132-236.ec2.internal\" (UID: \"18bcd73158ca1a5b1c82fc4d26d8e309\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.693899 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.693875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/505b89dd4f29cd1d6366a8221bca889f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal\" (UID: \"505b89dd4f29cd1d6366a8221bca889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.700583 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.700570 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:46.712766 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.712748 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-236.ec2.internal\" not found" node="ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.717258 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.717245 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-236.ec2.internal\" not found" node="ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.794908 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.794888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/505b89dd4f29cd1d6366a8221bca889f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal\" (UID: \"505b89dd4f29cd1d6366a8221bca889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.794970 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.794911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/505b89dd4f29cd1d6366a8221bca889f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal\" (UID: \"505b89dd4f29cd1d6366a8221bca889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.794970 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.794928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18bcd73158ca1a5b1c82fc4d26d8e309-config\") pod \"kube-apiserver-proxy-ip-10-0-132-236.ec2.internal\" (UID: \"18bcd73158ca1a5b1c82fc4d26d8e309\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.795035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.794973 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/505b89dd4f29cd1d6366a8221bca889f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal\" (UID: \"505b89dd4f29cd1d6366a8221bca889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.795035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.794978 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18bcd73158ca1a5b1c82fc4d26d8e309-config\") pod \"kube-apiserver-proxy-ip-10-0-132-236.ec2.internal\" (UID: \"18bcd73158ca1a5b1c82fc4d26d8e309\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.795035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:46.794988 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/505b89dd4f29cd1d6366a8221bca889f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal\" (UID: \"505b89dd4f29cd1d6366a8221bca889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" Jul 07 13:27:46.801283 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.801264 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:46.902107 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:46.902037 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:47.002497 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:47.002477 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:47.016646 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.016626 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" Jul 07 13:27:47.020919 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.020906 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal" Jul 07 13:27:47.102955 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:47.102927 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:47.203466 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:47.203397 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:47.303924 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:47.303890 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:47.332062 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.332036 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 13:27:47.332490 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.332194 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 13:27:47.404739 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:47.404709 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-236.ec2.internal\" not found" Jul 07 13:27:47.412295 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.412267 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 13:22:46 +0000 UTC" deadline="2028-02-07 10:45:03.483092988 +0000 UTC" Jul 07 13:27:47.412295 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.412293 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13917h17m16.070802178s" Jul 07 13:27:47.435589 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.435572 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 13:27:47.492918 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.492864 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal" Jul 07 13:27:47.505988 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.505964 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 13:27:47.508027 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.508014 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" Jul 07 13:27:47.525897 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.525877 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 13:27:47.684347 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:47.684313 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505b89dd4f29cd1d6366a8221bca889f.slice/crio-8f5067a61ae9e39d09d7a012f67d4634ff54179490b94dcefba721fdc58e19f8 WatchSource:0}: Error finding container 8f5067a61ae9e39d09d7a012f67d4634ff54179490b94dcefba721fdc58e19f8: Status 404 returned error can't find the container with id 8f5067a61ae9e39d09d7a012f67d4634ff54179490b94dcefba721fdc58e19f8 Jul 07 13:27:47.689988 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.689972 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 13:27:47.782246 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:47.782219 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bcd73158ca1a5b1c82fc4d26d8e309.slice/crio-55aceaa7e46e0531e880bcc5d644275d46fafcc49847b0c7e2d75b535dbbf990 WatchSource:0}: Error finding container 55aceaa7e46e0531e880bcc5d644275d46fafcc49847b0c7e2d75b535dbbf990: Status 404 returned error can't find the container with id 55aceaa7e46e0531e880bcc5d644275d46fafcc49847b0c7e2d75b535dbbf990 Jul 07 13:27:47.810068 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.810046 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 13:27:47.959623 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:47.959597 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 13:27:48.370655 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.370625 2576 apiserver.go:52] "Watching apiserver" Jul 07 13:27:48.378827 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.378802 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 13:27:48.380096 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.380069 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-t57fq","openshift-multus/multus-mbwql","openshift-network-operator/iptables-alerter-6kww5","kube-system/global-pull-secret-syncer-dcg6d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj","openshift-cluster-node-tuning-operator/tuned-w7t7p","openshift-dns/node-resolver-qhhwm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal","openshift-multus/network-metrics-daemon-pgw8q","openshift-network-diagnostics/network-check-target-tq7cd","openshift-ovn-kubernetes/ovnkube-node-qnr4d","kube-system/konnectivity-agent-ns4bt","kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal","openshift-image-registry/node-ca-wcqgp"] Jul 07 13:27:48.381620 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.381598 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.383320 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.383292 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.384411 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.384394 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.386091 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.385994 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:48.386091 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.386057 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:27:48.388066 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.387747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.389287 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.389004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.392935 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.392914 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 13:27:48.393193 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.393157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:48.393279 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.393242 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:27:48.394858 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.394837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.397927 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.397906 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 13:27:48.398175 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.398145 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 13:27:48.398433 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.398418 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ltq29\"" Jul 07 13:27:48.399410 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.399394 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.400362 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.400344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:48.400436 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.400405 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:27:48.401666 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.401645 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.403376 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5e5fa67-7403-4ef6-920d-d3f15c658361-env-overrides\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.403470 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32f0bc68-c8ce-4756-8944-97f48df4e7cb-dbus\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:48.403531 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-device-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.403531 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-sysconfig\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.403628 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-sysctl-conf\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.403628 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-node-log\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.403628 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-kubernetes\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.403628 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-run\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.403808 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-etc-selinux\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.403808 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-run-systemd\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.403808 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb79r\" (UniqueName: \"kubernetes.io/projected/28b8a5de-4b31-4f32-b6b4-4e118987db04-kube-api-access-qb79r\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:48.403808 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-cni-bin\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.403808 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5e5fa67-7403-4ef6-920d-d3f15c658361-ovnkube-config\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.403808 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17a9a9ab-5b3b-4805-b971-3be2d30b46ec-host\") pod \"node-ca-wcqgp\" (UID: \"17a9a9ab-5b3b-4805-b971-3be2d30b46ec\") " pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.403808 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17a9a9ab-5b3b-4805-b971-3be2d30b46ec-serviceca\") pod \"node-ca-wcqgp\" (UID: \"17a9a9ab-5b3b-4805-b971-3be2d30b46ec\") " pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.403808 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-log-socket\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32f0bc68-c8ce-4756-8944-97f48df4e7cb-kubelet-config\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-sys-fs\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-modprobe-d\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0165e63b-34ba-494b-9a7b-31eb2e652b54-tmp-dir\") pod \"node-resolver-qhhwm\" (UID: \"0165e63b-34ba-494b-9a7b-31eb2e652b54\") " pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdtm\" (UniqueName: \"kubernetes.io/projected/17a9a9ab-5b3b-4805-b971-3be2d30b46ec-kube-api-access-9hdtm\") pod \"node-ca-wcqgp\" (UID: \"17a9a9ab-5b3b-4805-b971-3be2d30b46ec\") " pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-tmp\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.403985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw5jp\" (UniqueName: \"kubernetes.io/projected/871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb-kube-api-access-kw5jp\") pod \"iptables-alerter-6kww5\" (UID: \"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb\") " pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-kubelet\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-registration-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-cni-netd\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7thb\" (UniqueName: \"kubernetes.io/projected/e5e5fa67-7403-4ef6-920d-d3f15c658361-kube-api-access-b7thb\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-sysctl-d\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-host\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-socket-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.404222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5e5fa67-7403-4ef6-920d-d3f15c658361-ovnkube-script-lib\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-systemd-units\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0165e63b-34ba-494b-9a7b-31eb2e652b54-hosts-file\") pod \"node-resolver-qhhwm\" (UID: \"0165e63b-34ba-494b-9a7b-31eb2e652b54\") " pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb-iptables-alerter-script\") pod \"iptables-alerter-6kww5\" (UID: \"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb\") " pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnkq4\" (UniqueName: \"kubernetes.io/projected/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-kube-api-access-bnkq4\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-systemd\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-sys\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-lib-modules\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-var-lib-kubelet\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-tuned\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-var-lib-openvswitch\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjdk\" (UniqueName: \"kubernetes.io/projected/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-kube-api-access-hwjdk\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh4zn\" (UniqueName: \"kubernetes.io/projected/0165e63b-34ba-494b-9a7b-31eb2e652b54-kube-api-access-qh4zn\") pod \"node-resolver-qhhwm\" (UID: \"0165e63b-34ba-494b-9a7b-31eb2e652b54\") " pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.404952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-slash\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.405666 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-run-netns\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.405666 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb-host-slash\") pod \"iptables-alerter-6kww5\" (UID: \"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb\") " pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.405666 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5e5fa67-7403-4ef6-920d-d3f15c658361-ovn-node-metrics-cert\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.405666 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:48.405666 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-run-openvswitch\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.405666 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-etc-openvswitch\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.405666 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.404957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-run-ovn\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.406216 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.406180 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 13:27:48.406308 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.406272 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 13:27:48.408369 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.408260 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qlnhp\"" Jul 07 13:27:48.408369 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.408307 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 13:27:48.408369 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.408322 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 13:27:48.408369 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.408344 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 13:27:48.408651 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.408531 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 13:27:48.408651 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.408545 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 13:27:48.408651 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.408584 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 13:27:48.408651 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.408641 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 13:27:48.408835 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.408711 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 13:27:48.408835 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.408793 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 13:27:48.409899 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.409825 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 13:27:48.409899 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.409891 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 13:27:48.410010 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.409929 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 13:27:48.410010 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.410001 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 13:27:48.410184 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.410153 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 13:27:48.411018 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.410653 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 13:27:48.411018 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.410843 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 13:27:48.411018 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.411011 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 13:27:48.411228 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.411047 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 13:27:48.426799 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.426777 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p67xp\"" Jul 07 13:27:48.427782 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.427763 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 13:27:48.429382 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.429351 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-42sgz\"" Jul 07 13:27:48.429477 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.429396 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-72wt6\"" Jul 07 13:27:48.429677 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.429543 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6ld8h\"" Jul 07 13:27:48.429677 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.429616 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 13:27:48.432961 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.432942 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v2jwt\"" Jul 07 13:27:48.442203 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.441570 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 13:27:48.442203 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.441733 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-84fpw\"" Jul 07 13:27:48.442203 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.441818 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 13:27:48.442203 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.442017 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xxlr6\"" Jul 07 13:27:48.442203 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.442100 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 13:27:48.442203 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.442137 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 13:27:48.460330 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.460098 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 13:27:48.467610 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.467585 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9g5b9" Jul 07 13:27:48.488729 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.488711 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9g5b9" Jul 07 13:27:48.494541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.494523 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 13:27:48.505691 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-hostroot\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.505775 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48kvd\" (UniqueName: \"kubernetes.io/projected/555fe057-e322-461c-9391-cfc0d2596f6f-kube-api-access-48kvd\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.505775 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-var-lib-kubelet\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.505922 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505799 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-conf-dir\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.505922 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/555fe057-e322-461c-9391-cfc0d2596f6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.505922 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-kubelet\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.505922 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-registration-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.505922 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-etc-kubernetes\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4edb049-06f0-4602-9c6f-592e4d35ca14-konnectivity-ca\") pod \"konnectivity-agent-ns4bt\" (UID: \"b4edb049-06f0-4602-9c6f-592e4d35ca14\") " pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505958 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-kubelet\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.505968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-cni-netd\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-cni-netd\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506044 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7thb\" (UniqueName: \"kubernetes.io/projected/e5e5fa67-7403-4ef6-920d-d3f15c658361-kube-api-access-b7thb\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-sysctl-d\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-host\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-cnibin\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-os-release\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506144 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-registration-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.506207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-socket-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-system-cni-dir\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506251 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-host\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/555fe057-e322-461c-9391-cfc0d2596f6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5e5fa67-7403-4ef6-920d-d3f15c658361-ovnkube-script-lib\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-systemd-units\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0165e63b-34ba-494b-9a7b-31eb2e652b54-hosts-file\") pod \"node-resolver-qhhwm\" (UID: \"0165e63b-34ba-494b-9a7b-31eb2e652b54\") " pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb-iptables-alerter-script\") pod \"iptables-alerter-6kww5\" (UID: \"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb\") " pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506422 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-socket-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506427 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnkq4\" (UniqueName: \"kubernetes.io/projected/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-kube-api-access-bnkq4\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.506532 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-systemd\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-sys\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.506597 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret podName:32f0bc68-c8ce-4756-8944-97f48df4e7cb nodeName:}" failed. No retries permitted until 2026-07-07 13:27:49.00657771 +0000 UTC m=+3.007819232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret") pod "global-pull-secret-syncer-dcg6d" (UID: "32f0bc68-c8ce-4756-8944-97f48df4e7cb") : object "kube-system"/"original-pull-secret" not registered Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-lib-modules\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.506716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506633 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-sys\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-var-lib-kubelet\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-tuned\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-var-lib-openvswitch\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjdk\" (UniqueName: \"kubernetes.io/projected/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-kube-api-access-hwjdk\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-cni-dir\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh4zn\" (UniqueName: \"kubernetes.io/projected/0165e63b-34ba-494b-9a7b-31eb2e652b54-kube-api-access-qh4zn\") pod \"node-resolver-qhhwm\" (UID: \"0165e63b-34ba-494b-9a7b-31eb2e652b54\") " pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr6kr\" (UniqueName: \"kubernetes.io/projected/72ed650a-018b-4c84-ac59-71f6e2227f0e-kube-api-access-xr6kr\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/555fe057-e322-461c-9391-cfc0d2596f6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-slash\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-run-netns\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb-host-slash\") pod \"iptables-alerter-6kww5\" (UID: \"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb\") " pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-var-lib-cni-multus\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4edb049-06f0-4602-9c6f-592e4d35ca14-agent-certs\") pod \"konnectivity-agent-ns4bt\" (UID: \"b4edb049-06f0-4602-9c6f-592e4d35ca14\") " pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5e5fa67-7403-4ef6-920d-d3f15c658361-ovn-node-metrics-cert\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507033 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5e5fa67-7403-4ef6-920d-d3f15c658361-ovnkube-script-lib\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.507541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-run-openvswitch\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-run-k8s-cni-cncf-io\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-var-lib-cni-bin\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507150 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqx4\" (UniqueName: \"kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4\") pod \"network-check-target-tq7cd\" (UID: \"bff31dac-b32a-4258-b236-2d5c0c58adc6\") " pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-etc-openvswitch\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507038 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-systemd\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507280 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-slash\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-run-ovn\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507320 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-run-ovn\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507331 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-systemd-units\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5e5fa67-7403-4ef6-920d-d3f15c658361-env-overrides\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507463 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-lib-modules\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.506376 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-sysctl-d\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507524 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-var-lib-openvswitch\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507503 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0165e63b-34ba-494b-9a7b-31eb2e652b54-hosts-file\") pod \"node-resolver-qhhwm\" (UID: \"0165e63b-34ba-494b-9a7b-31eb2e652b54\") " pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507557 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-var-lib-kubelet\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.508307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507580 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-etc-openvswitch\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32f0bc68-c8ce-4756-8944-97f48df4e7cb-dbus\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-device-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-sysconfig\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507646 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb-iptables-alerter-script\") pod \"iptables-alerter-6kww5\" (UID: \"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb\") " pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-sysctl-conf\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.507674 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-socket-dir-parent\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507712 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-run-netns\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507737 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.507772 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs podName:28b8a5de-4b31-4f32-b6b4-4e118987db04 nodeName:}" failed. No retries permitted until 2026-07-07 13:27:49.007706374 +0000 UTC m=+3.008947893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs") pod "network-metrics-daemon-pgw8q" (UID: "28b8a5de-4b31-4f32-b6b4-4e118987db04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507832 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32f0bc68-c8ce-4756-8944-97f48df4e7cb-dbus\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507874 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-sysctl-conf\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-node-log\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-kubernetes\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-run\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.508983 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508009 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5e5fa67-7403-4ef6-920d-d3f15c658361-env-overrides\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-os-release\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.507594 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb-host-slash\") pod \"iptables-alerter-6kww5\" (UID: \"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb\") " pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508082 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-node-log\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-etc-selinux\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508189 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-device-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508201 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-kubernetes\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508200 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-run-openvswitch\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-run-systemd\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508242 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-etc-selinux\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb79r\" (UniqueName: \"kubernetes.io/projected/28b8a5de-4b31-4f32-b6b4-4e118987db04-kube-api-access-qb79r\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508276 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-sysconfig\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-cni-bin\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508230 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508306 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-run\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508319 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-run-systemd\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5e5fa67-7403-4ef6-920d-d3f15c658361-ovnkube-config\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.509756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17a9a9ab-5b3b-4805-b971-3be2d30b46ec-host\") pod \"node-ca-wcqgp\" (UID: \"17a9a9ab-5b3b-4805-b971-3be2d30b46ec\") " pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508471 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-host-cni-bin\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17a9a9ab-5b3b-4805-b971-3be2d30b46ec-serviceca\") pod \"node-ca-wcqgp\" (UID: \"17a9a9ab-5b3b-4805-b971-3be2d30b46ec\") " pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508531 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17a9a9ab-5b3b-4805-b971-3be2d30b46ec-host\") pod \"node-ca-wcqgp\" (UID: \"17a9a9ab-5b3b-4805-b971-3be2d30b46ec\") " pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-system-cni-dir\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-log-socket\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32f0bc68-c8ce-4756-8944-97f48df4e7cb-kubelet-config\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5e5fa67-7403-4ef6-920d-d3f15c658361-log-socket\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-sys-fs\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508775 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32f0bc68-c8ce-4756-8944-97f48df4e7cb-kubelet-config\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-run-netns\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-daemon-config\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508868 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-sys-fs\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-run-multus-certs\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508888 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5e5fa67-7403-4ef6-920d-d3f15c658361-ovnkube-config\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-cnibin\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.510543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508952 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17a9a9ab-5b3b-4805-b971-3be2d30b46ec-serviceca\") pod \"node-ca-wcqgp\" (UID: \"17a9a9ab-5b3b-4805-b971-3be2d30b46ec\") " pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.511341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.508983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-modprobe-d\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.511341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.509019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0165e63b-34ba-494b-9a7b-31eb2e652b54-tmp-dir\") pod \"node-resolver-qhhwm\" (UID: \"0165e63b-34ba-494b-9a7b-31eb2e652b54\") " pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.511341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.509071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdtm\" (UniqueName: \"kubernetes.io/projected/17a9a9ab-5b3b-4805-b971-3be2d30b46ec-kube-api-access-9hdtm\") pod \"node-ca-wcqgp\" (UID: \"17a9a9ab-5b3b-4805-b971-3be2d30b46ec\") " pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.511341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.509090 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-modprobe-d\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.511341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.509115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.511341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.509236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-tmp\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.511341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.509267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5jp\" (UniqueName: \"kubernetes.io/projected/871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb-kube-api-access-kw5jp\") pod \"iptables-alerter-6kww5\" (UID: \"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb\") " pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.511341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.509291 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0165e63b-34ba-494b-9a7b-31eb2e652b54-tmp-dir\") pod \"node-resolver-qhhwm\" (UID: \"0165e63b-34ba-494b-9a7b-31eb2e652b54\") " pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.511341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.509337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72ed650a-018b-4c84-ac59-71f6e2227f0e-cni-binary-copy\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.511341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.509530 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.511809 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.511616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-etc-tuned\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.511809 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.511665 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-tmp\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.513342 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.513324 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5e5fa67-7403-4ef6-920d-d3f15c658361-ovn-node-metrics-cert\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.519903 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.519881 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7thb\" (UniqueName: \"kubernetes.io/projected/e5e5fa67-7403-4ef6-920d-d3f15c658361-kube-api-access-b7thb\") pod \"ovnkube-node-qnr4d\" (UID: \"e5e5fa67-7403-4ef6-920d-d3f15c658361\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.533652 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.532685 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb79r\" (UniqueName: \"kubernetes.io/projected/28b8a5de-4b31-4f32-b6b4-4e118987db04-kube-api-access-qb79r\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:48.533652 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.533402 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh4zn\" (UniqueName: \"kubernetes.io/projected/0165e63b-34ba-494b-9a7b-31eb2e652b54-kube-api-access-qh4zn\") pod \"node-resolver-qhhwm\" (UID: \"0165e63b-34ba-494b-9a7b-31eb2e652b54\") " pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.534573 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.534509 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnkq4\" (UniqueName: \"kubernetes.io/projected/5f39c3f4-9ec0-47ad-bc29-bdd7be431470-kube-api-access-bnkq4\") pod \"tuned-w7t7p\" (UID: \"5f39c3f4-9ec0-47ad-bc29-bdd7be431470\") " pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.536604 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.536561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdtm\" (UniqueName: \"kubernetes.io/projected/17a9a9ab-5b3b-4805-b971-3be2d30b46ec-kube-api-access-9hdtm\") pod \"node-ca-wcqgp\" (UID: \"17a9a9ab-5b3b-4805-b971-3be2d30b46ec\") " pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.536695 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.536625 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5jp\" (UniqueName: \"kubernetes.io/projected/871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb-kube-api-access-kw5jp\") pod \"iptables-alerter-6kww5\" (UID: \"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb\") " pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.537946 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.537920 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjdk\" (UniqueName: \"kubernetes.io/projected/6b7a7c00-45c0-4cc6-8ff2-000485810e8d-kube-api-access-hwjdk\") pod \"aws-ebs-csi-driver-node-48jrj\" (UID: \"6b7a7c00-45c0-4cc6-8ff2-000485810e8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.591053 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.591008 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" event={"ID":"505b89dd4f29cd1d6366a8221bca889f","Type":"ContainerStarted","Data":"8f5067a61ae9e39d09d7a012f67d4634ff54179490b94dcefba721fdc58e19f8"} Jul 07 13:27:48.592297 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.592248 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal" event={"ID":"18bcd73158ca1a5b1c82fc4d26d8e309","Type":"ContainerStarted","Data":"55aceaa7e46e0531e880bcc5d644275d46fafcc49847b0c7e2d75b535dbbf990"} Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-cnibin\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-os-release\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-system-cni-dir\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610222 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-cnibin\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/555fe057-e322-461c-9391-cfc0d2596f6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610250 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-os-release\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-cni-dir\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr6kr\" (UniqueName: \"kubernetes.io/projected/72ed650a-018b-4c84-ac59-71f6e2227f0e-kube-api-access-xr6kr\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/555fe057-e322-461c-9391-cfc0d2596f6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610373 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-system-cni-dir\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-var-lib-cni-multus\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610433 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-var-lib-cni-multus\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610627 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-cni-dir\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4edb049-06f0-4602-9c6f-592e4d35ca14-agent-certs\") pod \"konnectivity-agent-ns4bt\" (UID: \"b4edb049-06f0-4602-9c6f-592e4d35ca14\") " pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-run-k8s-cni-cncf-io\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-var-lib-cni-bin\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqx4\" (UniqueName: \"kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4\") pod \"network-check-target-tq7cd\" (UID: \"bff31dac-b32a-4258-b236-2d5c0c58adc6\") " pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:48.610846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-socket-dir-parent\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610779 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/555fe057-e322-461c-9391-cfc0d2596f6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-os-release\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610830 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-var-lib-cni-bin\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610915 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/555fe057-e322-461c-9391-cfc0d2596f6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610974 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-run-k8s-cni-cncf-io\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.610832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-system-cni-dir\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-run-netns\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611030 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-socket-dir-parent\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-daemon-config\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-run-multus-certs\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-cnibin\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611130 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-system-cni-dir\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72ed650a-018b-4c84-ac59-71f6e2227f0e-cni-binary-copy\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611093 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-os-release\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611184 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-run-multus-certs\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.611774 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-cnibin\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611320 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/555fe057-e322-461c-9391-cfc0d2596f6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611360 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-run-netns\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-hostroot\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48kvd\" (UniqueName: \"kubernetes.io/projected/555fe057-e322-461c-9391-cfc0d2596f6f-kube-api-access-48kvd\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-var-lib-kubelet\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-conf-dir\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/555fe057-e322-461c-9391-cfc0d2596f6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-etc-kubernetes\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4edb049-06f0-4602-9c6f-592e4d35ca14-konnectivity-ca\") pod \"konnectivity-agent-ns4bt\" (UID: \"b4edb049-06f0-4602-9c6f-592e4d35ca14\") " pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611715 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-daemon-config\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72ed650a-018b-4c84-ac59-71f6e2227f0e-cni-binary-copy\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-host-var-lib-kubelet\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611777 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-hostroot\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-multus-conf-dir\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611862 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ed650a-018b-4c84-ac59-71f6e2227f0e-etc-kubernetes\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.611930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4edb049-06f0-4602-9c6f-592e4d35ca14-konnectivity-ca\") pod \"konnectivity-agent-ns4bt\" (UID: \"b4edb049-06f0-4602-9c6f-592e4d35ca14\") " pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:27:48.612580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.612219 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/555fe057-e322-461c-9391-cfc0d2596f6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.613527 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.613487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4edb049-06f0-4602-9c6f-592e4d35ca14-agent-certs\") pod \"konnectivity-agent-ns4bt\" (UID: \"b4edb049-06f0-4602-9c6f-592e4d35ca14\") " pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:27:48.632064 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.632004 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 13:27:48.632064 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.632030 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 13:27:48.632064 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.632045 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2hqx4 for pod openshift-network-diagnostics/network-check-target-tq7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:27:48.632348 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:48.632106 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4 podName:bff31dac-b32a-4258-b236-2d5c0c58adc6 nodeName:}" failed. No retries permitted until 2026-07-07 13:27:49.132089848 +0000 UTC m=+3.133331373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2hqx4" (UniqueName: "kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4") pod "network-check-target-tq7cd" (UID: "bff31dac-b32a-4258-b236-2d5c0c58adc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:27:48.633601 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.633558 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr6kr\" (UniqueName: \"kubernetes.io/projected/72ed650a-018b-4c84-ac59-71f6e2227f0e-kube-api-access-xr6kr\") pod \"multus-mbwql\" (UID: \"72ed650a-018b-4c84-ac59-71f6e2227f0e\") " pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.634187 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.634138 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48kvd\" (UniqueName: \"kubernetes.io/projected/555fe057-e322-461c-9391-cfc0d2596f6f-kube-api-access-48kvd\") pod \"multus-additional-cni-plugins-t57fq\" (UID: \"555fe057-e322-461c-9391-cfc0d2596f6f\") " pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.698250 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.698207 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qhhwm" Jul 07 13:27:48.706237 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.706213 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:27:48.715783 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.715758 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6kww5" Jul 07 13:27:48.721739 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.721719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" Jul 07 13:27:48.728850 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.728835 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" Jul 07 13:27:48.737160 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.737142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wcqgp" Jul 07 13:27:48.743853 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.743831 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mbwql" Jul 07 13:27:48.750612 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.750593 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t57fq" Jul 07 13:27:48.755937 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:48.755919 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:27:49.013411 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:49.013331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:49.013411 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:49.013384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:49.013618 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:49.013497 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:27:49.013618 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:49.013501 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 13:27:49.013618 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:49.013562 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs podName:28b8a5de-4b31-4f32-b6b4-4e118987db04 nodeName:}" failed. No retries permitted until 2026-07-07 13:27:50.013546135 +0000 UTC m=+4.014787639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs") pod "network-metrics-daemon-pgw8q" (UID: "28b8a5de-4b31-4f32-b6b4-4e118987db04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:27:49.013618 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:49.013579 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret podName:32f0bc68-c8ce-4756-8944-97f48df4e7cb nodeName:}" failed. No retries permitted until 2026-07-07 13:27:50.013570869 +0000 UTC m=+4.014812378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret") pod "global-pull-secret-syncer-dcg6d" (UID: "32f0bc68-c8ce-4756-8944-97f48df4e7cb") : object "kube-system"/"original-pull-secret" not registered Jul 07 13:27:49.214823 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:49.214786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqx4\" (UniqueName: \"kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4\") pod \"network-check-target-tq7cd\" (UID: \"bff31dac-b32a-4258-b236-2d5c0c58adc6\") " pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:49.214999 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:49.214968 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 13:27:49.214999 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:49.214991 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 13:27:49.214999 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:49.215001 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2hqx4 for pod openshift-network-diagnostics/network-check-target-tq7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:27:49.215119 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:49.215058 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4 podName:bff31dac-b32a-4258-b236-2d5c0c58adc6 nodeName:}" failed. No retries permitted until 2026-07-07 13:27:50.215038799 +0000 UTC m=+4.216280314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hqx4" (UniqueName: "kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4") pod "network-check-target-tq7cd" (UID: "bff31dac-b32a-4258-b236-2d5c0c58adc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:27:49.490384 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:49.490276 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 13:22:48 +0000 UTC" deadline="2028-02-08 06:06:05.873809999 +0000 UTC" Jul 07 13:27:49.490384 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:49.490338 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13936h38m16.383475931s" Jul 07 13:27:49.583789 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:49.583749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:49.583958 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:49.583867 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:27:49.621571 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:49.621549 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0165e63b_34ba_494b_9a7b_31eb2e652b54.slice/crio-81c27a3fbeec6cc5881ba20c825acdbdbed5181372869abf6a4f39624c5189f3 WatchSource:0}: Error finding container 81c27a3fbeec6cc5881ba20c825acdbdbed5181372869abf6a4f39624c5189f3: Status 404 returned error can't find the container with id 81c27a3fbeec6cc5881ba20c825acdbdbed5181372869abf6a4f39624c5189f3 Jul 07 13:27:49.623764 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:49.623398 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e5fa67_7403_4ef6_920d_d3f15c658361.slice/crio-bc1f9460ee6072c3005b2ae46cc33486b07b6b8243de32d93cb4975bad3ae5ec WatchSource:0}: Error finding container bc1f9460ee6072c3005b2ae46cc33486b07b6b8243de32d93cb4975bad3ae5ec: Status 404 returned error can't find the container with id bc1f9460ee6072c3005b2ae46cc33486b07b6b8243de32d93cb4975bad3ae5ec Jul 07 13:27:49.626808 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:49.626783 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f39c3f4_9ec0_47ad_bc29_bdd7be431470.slice/crio-ae2f55c84dc7c4c11ac1de5f3f3a922240b973c6d437897981a24c5516c2805d WatchSource:0}: Error finding container ae2f55c84dc7c4c11ac1de5f3f3a922240b973c6d437897981a24c5516c2805d: Status 404 returned error can't find the container with id ae2f55c84dc7c4c11ac1de5f3f3a922240b973c6d437897981a24c5516c2805d Jul 07 13:27:49.629279 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:49.629255 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555fe057_e322_461c_9391_cfc0d2596f6f.slice/crio-83a938582f6c898e320f22028a004795470a3defe9dab91455304b669d0d7ebe WatchSource:0}: Error finding container 83a938582f6c898e320f22028a004795470a3defe9dab91455304b669d0d7ebe: Status 404 returned error can't find the container with id 83a938582f6c898e320f22028a004795470a3defe9dab91455304b669d0d7ebe Jul 07 13:27:49.630558 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:49.630342 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a9a9ab_5b3b_4805_b971_3be2d30b46ec.slice/crio-d9ff9b1ad79c57bf2ca6aaf9406b099304c9006eece5ac54505009bb68f0da64 WatchSource:0}: Error finding container d9ff9b1ad79c57bf2ca6aaf9406b099304c9006eece5ac54505009bb68f0da64: Status 404 returned error can't find the container with id d9ff9b1ad79c57bf2ca6aaf9406b099304c9006eece5ac54505009bb68f0da64 Jul 07 13:27:49.631336 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:27:49.631318 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ed650a_018b_4c84_ac59_71f6e2227f0e.slice/crio-669715d22bfcf326351b3d9f8d10ef931c69f6ec0ab4818ce7fe25185f11dc6a WatchSource:0}: Error finding container 669715d22bfcf326351b3d9f8d10ef931c69f6ec0ab4818ce7fe25185f11dc6a: Status 404 returned error can't find the container with id 669715d22bfcf326351b3d9f8d10ef931c69f6ec0ab4818ce7fe25185f11dc6a Jul 07 13:27:50.019905 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.019669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:50.019905 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:50.019904 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 13:27:50.020122 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.019942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:50.020122 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:50.019974 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret podName:32f0bc68-c8ce-4756-8944-97f48df4e7cb nodeName:}" failed. No retries permitted until 2026-07-07 13:27:52.01995365 +0000 UTC m=+6.021195174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret") pod "global-pull-secret-syncer-dcg6d" (UID: "32f0bc68-c8ce-4756-8944-97f48df4e7cb") : object "kube-system"/"original-pull-secret" not registered Jul 07 13:27:50.020122 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:50.020031 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:27:50.020122 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:50.020104 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs podName:28b8a5de-4b31-4f32-b6b4-4e118987db04 nodeName:}" failed. No retries permitted until 2026-07-07 13:27:52.020063775 +0000 UTC m=+6.021305282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs") pod "network-metrics-daemon-pgw8q" (UID: "28b8a5de-4b31-4f32-b6b4-4e118987db04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:27:50.221541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.221507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqx4\" (UniqueName: \"kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4\") pod \"network-check-target-tq7cd\" (UID: \"bff31dac-b32a-4258-b236-2d5c0c58adc6\") " pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:50.221723 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:50.221707 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 13:27:50.221791 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:50.221730 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 13:27:50.221791 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:50.221743 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2hqx4 for pod openshift-network-diagnostics/network-check-target-tq7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:27:50.221903 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:50.221818 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4 podName:bff31dac-b32a-4258-b236-2d5c0c58adc6 nodeName:}" failed. No retries permitted until 2026-07-07 13:27:52.2217993 +0000 UTC m=+6.223040821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hqx4" (UniqueName: "kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4") pod "network-check-target-tq7cd" (UID: "bff31dac-b32a-4258-b236-2d5c0c58adc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:27:50.491355 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.491266 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 13:22:48 +0000 UTC" deadline="2028-01-22 01:41:37.002035961 +0000 UTC" Jul 07 13:27:50.491355 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.491306 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13524h13m46.510733684s" Jul 07 13:27:50.586927 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.586899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:50.587093 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:50.587030 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:27:50.587445 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.587422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:50.587541 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:50.587519 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:27:50.608158 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.608129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal" event={"ID":"18bcd73158ca1a5b1c82fc4d26d8e309","Type":"ContainerStarted","Data":"e3ce7094c60010662d98f7ed082304d7b6cebb756dbcecc732237a5b57fb17d7"} Jul 07 13:27:50.617028 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.616992 2576 generic.go:356] "Generic (PLEG): container finished" podID="505b89dd4f29cd1d6366a8221bca889f" containerID="41765f5257a86a70147263a18bc982bee873db4b56fc732525492aeefe3660c9" exitCode=0 Jul 07 13:27:50.617277 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.617253 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" event={"ID":"505b89dd4f29cd1d6366a8221bca889f","Type":"ContainerDied","Data":"41765f5257a86a70147263a18bc982bee873db4b56fc732525492aeefe3660c9"} Jul 07 13:27:50.624931 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.624904 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6kww5" event={"ID":"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb","Type":"ContainerStarted","Data":"e1fa48bcf233fdb884e6fad0fb3882f0dc9349d1c9aa4e1fbd37054dec7039b5"} Jul 07 13:27:50.634991 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.634959 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" event={"ID":"6b7a7c00-45c0-4cc6-8ff2-000485810e8d","Type":"ContainerStarted","Data":"aa236deb83c38c89b0cb4e14188a76b246c4b9389f188d1067ce3dc4de3bf5e9"} Jul 07 13:27:50.646714 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.646307 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-236.ec2.internal" podStartSLOduration=3.646290622 podStartE2EDuration="3.646290622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:27:50.624631184 +0000 UTC m=+4.625872712" watchObservedRunningTime="2026-07-07 13:27:50.646290622 +0000 UTC m=+4.647532151" Jul 07 13:27:50.650102 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.650028 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wcqgp" event={"ID":"17a9a9ab-5b3b-4805-b971-3be2d30b46ec","Type":"ContainerStarted","Data":"d9ff9b1ad79c57bf2ca6aaf9406b099304c9006eece5ac54505009bb68f0da64"} Jul 07 13:27:50.657717 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.657693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t57fq" event={"ID":"555fe057-e322-461c-9391-cfc0d2596f6f","Type":"ContainerStarted","Data":"83a938582f6c898e320f22028a004795470a3defe9dab91455304b669d0d7ebe"} Jul 07 13:27:50.662644 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.662139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" event={"ID":"5f39c3f4-9ec0-47ad-bc29-bdd7be431470","Type":"ContainerStarted","Data":"ae2f55c84dc7c4c11ac1de5f3f3a922240b973c6d437897981a24c5516c2805d"} Jul 07 13:27:50.671069 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.671046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" event={"ID":"e5e5fa67-7403-4ef6-920d-d3f15c658361","Type":"ContainerStarted","Data":"bc1f9460ee6072c3005b2ae46cc33486b07b6b8243de32d93cb4975bad3ae5ec"} Jul 07 13:27:50.683220 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.683193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ns4bt" event={"ID":"b4edb049-06f0-4602-9c6f-592e4d35ca14","Type":"ContainerStarted","Data":"d63547140ad1ce4d84230e3fb12a21f023c49528529ed55dda6973712d26a29d"} Jul 07 13:27:50.697611 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.697533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mbwql" event={"ID":"72ed650a-018b-4c84-ac59-71f6e2227f0e","Type":"ContainerStarted","Data":"669715d22bfcf326351b3d9f8d10ef931c69f6ec0ab4818ce7fe25185f11dc6a"} Jul 07 13:27:50.722066 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:50.722038 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qhhwm" event={"ID":"0165e63b-34ba-494b-9a7b-31eb2e652b54","Type":"ContainerStarted","Data":"81c27a3fbeec6cc5881ba20c825acdbdbed5181372869abf6a4f39624c5189f3"} Jul 07 13:27:51.584827 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:51.584350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:51.584827 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:51.584477 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:27:51.759291 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:51.759234 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" event={"ID":"505b89dd4f29cd1d6366a8221bca889f","Type":"ContainerStarted","Data":"c809cce116887bf75ca95f1560ab723a3c5a22fbe001750040cc36f4b8d0e770"} Jul 07 13:27:51.776927 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:51.776358 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-236.ec2.internal" podStartSLOduration=4.776337245 podStartE2EDuration="4.776337245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:27:51.776064385 +0000 UTC m=+5.777305904" watchObservedRunningTime="2026-07-07 13:27:51.776337245 +0000 UTC m=+5.777578775" Jul 07 13:27:52.039427 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:52.039340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:52.039427 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:52.039416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:52.039651 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:52.039478 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 13:27:52.039651 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:52.039554 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret podName:32f0bc68-c8ce-4756-8944-97f48df4e7cb nodeName:}" failed. No retries permitted until 2026-07-07 13:27:56.039534554 +0000 UTC m=+10.040776064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret") pod "global-pull-secret-syncer-dcg6d" (UID: "32f0bc68-c8ce-4756-8944-97f48df4e7cb") : object "kube-system"/"original-pull-secret" not registered Jul 07 13:27:52.039651 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:52.039583 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:27:52.039651 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:52.039640 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs podName:28b8a5de-4b31-4f32-b6b4-4e118987db04 nodeName:}" failed. No retries permitted until 2026-07-07 13:27:56.039622477 +0000 UTC m=+10.040864002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs") pod "network-metrics-daemon-pgw8q" (UID: "28b8a5de-4b31-4f32-b6b4-4e118987db04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:27:52.241759 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:52.241723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqx4\" (UniqueName: \"kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4\") pod \"network-check-target-tq7cd\" (UID: \"bff31dac-b32a-4258-b236-2d5c0c58adc6\") " pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:52.241914 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:52.241874 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 13:27:52.241914 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:52.241894 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 13:27:52.241914 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:52.241906 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2hqx4 for pod openshift-network-diagnostics/network-check-target-tq7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:27:52.242063 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:52.241965 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4 podName:bff31dac-b32a-4258-b236-2d5c0c58adc6 nodeName:}" failed. No retries permitted until 2026-07-07 13:27:56.241945829 +0000 UTC m=+10.243187348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hqx4" (UniqueName: "kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4") pod "network-check-target-tq7cd" (UID: "bff31dac-b32a-4258-b236-2d5c0c58adc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:27:52.586115 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:52.585378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:52.586115 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:52.585515 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:27:52.586115 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:52.585962 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:52.586115 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:52.586057 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:27:53.584096 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:53.583588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:53.584096 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:53.583756 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:27:54.586910 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:54.584760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:54.586910 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:54.584902 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:27:54.586910 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:54.585301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:54.586910 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:54.585392 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:27:55.583958 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:55.583928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:55.588102 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:55.584456 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:27:56.072276 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:56.071516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:56.072276 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:56.071603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:56.072276 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:56.071718 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 13:27:56.072276 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:56.071783 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret podName:32f0bc68-c8ce-4756-8944-97f48df4e7cb nodeName:}" failed. No retries permitted until 2026-07-07 13:28:04.07175863 +0000 UTC m=+18.073000137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret") pod "global-pull-secret-syncer-dcg6d" (UID: "32f0bc68-c8ce-4756-8944-97f48df4e7cb") : object "kube-system"/"original-pull-secret" not registered Jul 07 13:27:56.072276 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:56.072192 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:27:56.072276 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:56.072238 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs podName:28b8a5de-4b31-4f32-b6b4-4e118987db04 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:04.072222496 +0000 UTC m=+18.073464006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs") pod "network-metrics-daemon-pgw8q" (UID: "28b8a5de-4b31-4f32-b6b4-4e118987db04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:27:56.273760 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:56.273724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqx4\" (UniqueName: \"kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4\") pod \"network-check-target-tq7cd\" (UID: \"bff31dac-b32a-4258-b236-2d5c0c58adc6\") " pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:56.274212 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:56.273913 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 13:27:56.274212 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:56.273937 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 13:27:56.274212 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:56.273949 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2hqx4 for pod openshift-network-diagnostics/network-check-target-tq7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:27:56.274212 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:56.274008 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4 podName:bff31dac-b32a-4258-b236-2d5c0c58adc6 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:04.273991391 +0000 UTC m=+18.275232899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hqx4" (UniqueName: "kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4") pod "network-check-target-tq7cd" (UID: "bff31dac-b32a-4258-b236-2d5c0c58adc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:27:56.585257 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:56.584597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:56.585257 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:56.584710 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:27:56.585257 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:56.585096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:56.585257 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:56.585202 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:27:57.583704 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:57.583667 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:57.584159 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:57.583786 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:27:58.584414 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:58.584378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:27:58.584861 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:58.584387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:27:58.584861 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:58.584531 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:27:58.584861 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:58.584672 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:27:59.584195 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:27:59.584144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:27:59.584391 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:27:59.584303 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:00.584123 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:00.584045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:00.584572 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:00.584045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:00.584572 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:00.584194 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:00.584572 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:00.584298 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:01.583596 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:01.583558 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:01.583763 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:01.583714 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:02.584458 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:02.584422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:02.584913 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:02.584437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:02.584913 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:02.584573 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:02.584913 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:02.584599 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:03.583727 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:03.583686 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:03.583931 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:03.583844 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:04.133431 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:04.133396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:04.133737 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:04.133462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:04.133737 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:04.133533 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 13:28:04.133737 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:04.133590 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:28:04.133737 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:04.133600 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret podName:32f0bc68-c8ce-4756-8944-97f48df4e7cb nodeName:}" failed. No retries permitted until 2026-07-07 13:28:20.133584038 +0000 UTC m=+34.134825544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret") pod "global-pull-secret-syncer-dcg6d" (UID: "32f0bc68-c8ce-4756-8944-97f48df4e7cb") : object "kube-system"/"original-pull-secret" not registered Jul 07 13:28:04.133737 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:04.133639 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs podName:28b8a5de-4b31-4f32-b6b4-4e118987db04 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:20.133624301 +0000 UTC m=+34.134865806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs") pod "network-metrics-daemon-pgw8q" (UID: "28b8a5de-4b31-4f32-b6b4-4e118987db04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:28:04.334514 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:04.334457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqx4\" (UniqueName: \"kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4\") pod \"network-check-target-tq7cd\" (UID: \"bff31dac-b32a-4258-b236-2d5c0c58adc6\") " pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:04.335536 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:04.335010 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 13:28:04.335536 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:04.335044 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 13:28:04.335536 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:04.335067 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2hqx4 for pod openshift-network-diagnostics/network-check-target-tq7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:28:04.335536 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:04.335149 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4 podName:bff31dac-b32a-4258-b236-2d5c0c58adc6 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:20.335122955 +0000 UTC m=+34.336364481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hqx4" (UniqueName: "kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4") pod "network-check-target-tq7cd" (UID: "bff31dac-b32a-4258-b236-2d5c0c58adc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:28:04.584078 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:04.584047 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:04.584261 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:04.584194 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:04.584333 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:04.584259 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:04.584390 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:04.584369 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:05.583895 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:05.583858 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:05.584416 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:05.583991 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:06.585399 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:06.585362 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:06.585832 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:06.585491 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:06.585832 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:06.585543 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:06.585832 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:06.585658 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:07.584579 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:07.584545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:07.584742 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:07.584677 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:08.583920 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.583729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:08.584588 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:08.584036 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:08.584588 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.584482 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:08.584588 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:08.584580 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:08.792726 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.792700 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" event={"ID":"6b7a7c00-45c0-4cc6-8ff2-000485810e8d","Type":"ContainerStarted","Data":"52eb8ed4bb528e6dcfc33ede3d3e5a84522e7b9a48375b180ca40ee8ef3e19ce"} Jul 07 13:28:08.793953 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.793931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wcqgp" event={"ID":"17a9a9ab-5b3b-4805-b971-3be2d30b46ec","Type":"ContainerStarted","Data":"ca717b76874039b888b92e12322fbc627cc25bda20777c3ae0b94fa4886f11d9"} Jul 07 13:28:08.795357 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.795330 2576 generic.go:356] "Generic (PLEG): container finished" podID="555fe057-e322-461c-9391-cfc0d2596f6f" containerID="19ac926af710cd13376b3bcfef5d76623cb20e77cec6a0fabb8a5fb8c76444cf" exitCode=0 Jul 07 13:28:08.795454 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.795399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t57fq" event={"ID":"555fe057-e322-461c-9391-cfc0d2596f6f","Type":"ContainerDied","Data":"19ac926af710cd13376b3bcfef5d76623cb20e77cec6a0fabb8a5fb8c76444cf"} Jul 07 13:28:08.797014 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.796970 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" event={"ID":"5f39c3f4-9ec0-47ad-bc29-bdd7be431470","Type":"ContainerStarted","Data":"8bb988bec8f206132b607776512b7fa63d90ff0590a98876c9791d659270d1f8"} Jul 07 13:28:08.801723 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.801697 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" event={"ID":"e5e5fa67-7403-4ef6-920d-d3f15c658361","Type":"ContainerStarted","Data":"851ed4e496fe6b14e084492fa43ebbd34be7b236e4fac5fe753a80fedb21c368"} Jul 07 13:28:08.801805 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.801776 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" event={"ID":"e5e5fa67-7403-4ef6-920d-d3f15c658361","Type":"ContainerStarted","Data":"c0a9ef42f1bd1366a9bf45a44a4699b4fb600d10a5fe6ee38a16ebb77939aff8"} Jul 07 13:28:08.801873 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.801826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" event={"ID":"e5e5fa67-7403-4ef6-920d-d3f15c658361","Type":"ContainerStarted","Data":"3a2b7f2059395dd52399b2d41ca9b8f7f4d11eda9889e73096e0b3cdc136e78a"} Jul 07 13:28:08.801935 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.801875 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" event={"ID":"e5e5fa67-7403-4ef6-920d-d3f15c658361","Type":"ContainerStarted","Data":"ef89cee47c3d24b95ea5744d8536461daa2997bf91ec31c99bac14cb2df33901"} Jul 07 13:28:08.803007 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.802985 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ns4bt" event={"ID":"b4edb049-06f0-4602-9c6f-592e4d35ca14","Type":"ContainerStarted","Data":"89c60600eacaceaa1abcf540d96a31d094cf801aa73e919293c6d6d61ab30338"} Jul 07 13:28:08.804409 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.804381 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mbwql" event={"ID":"72ed650a-018b-4c84-ac59-71f6e2227f0e","Type":"ContainerStarted","Data":"0e373b09f2e232970d25217d4b08d9ab1e6a3fe960dbb27b45587bfb94fb7ba2"} Jul 07 13:28:08.805636 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.805596 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qhhwm" event={"ID":"0165e63b-34ba-494b-9a7b-31eb2e652b54","Type":"ContainerStarted","Data":"0328c4aa257ae9eacb2b7546dacbfa9f1709e2e49a4ac7e64009e60d222eb37a"} Jul 07 13:28:08.811636 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.811600 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wcqgp" podStartSLOduration=4.14379551 podStartE2EDuration="22.811588314s" totalImagesPullingTime="18.667792804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:08.810939747 +0000 UTC m=+22.812181273" watchObservedRunningTime="2026-07-07 13:28:08.811588314 +0000 UTC m=+22.812829841" Jul 07 13:28:08.834468 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.834431 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mbwql" podStartSLOduration=4.118035031 podStartE2EDuration="22.834420248s" totalImagesPullingTime="18.716385217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:08.834209102 +0000 UTC m=+22.835450626" watchObservedRunningTime="2026-07-07 13:28:08.834420248 +0000 UTC m=+22.835661774" Jul 07 13:28:08.880132 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.880095 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ns4bt" podStartSLOduration=8.369364947 podStartE2EDuration="22.880082593s" totalImagesPullingTime="14.510717646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:08.879663129 +0000 UTC m=+22.880904655" watchObservedRunningTime="2026-07-07 13:28:08.880082593 +0000 UTC m=+22.881324120" Jul 07 13:28:08.898039 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.897994 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w7t7p" podStartSLOduration=4.226434083 podStartE2EDuration="22.897981268s" totalImagesPullingTime="18.671547185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:08.897571862 +0000 UTC m=+22.898813402" watchObservedRunningTime="2026-07-07 13:28:08.897981268 +0000 UTC m=+22.899222794" Jul 07 13:28:08.913630 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:08.913596 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qhhwm" podStartSLOduration=4.235920832 podStartE2EDuration="22.913583715s" totalImagesPullingTime="18.677662883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:08.913228388 +0000 UTC m=+22.914469916" watchObservedRunningTime="2026-07-07 13:28:08.913583715 +0000 UTC m=+22.914825242" Jul 07 13:28:09.584328 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:09.584267 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:09.584675 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:09.584366 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:09.811236 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:09.811206 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" event={"ID":"e5e5fa67-7403-4ef6-920d-d3f15c658361","Type":"ContainerStarted","Data":"d76bfcde4133d2874892ab658ca41ef08eacb71b4a4b499adceaa9c49b3d89c8"} Jul 07 13:28:09.811377 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:09.811297 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" event={"ID":"e5e5fa67-7403-4ef6-920d-d3f15c658361","Type":"ContainerStarted","Data":"164960cba15c8e892c3174340eb57655e1cd579a2bd0083099d560c24f371f6a"} Jul 07 13:28:09.812749 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:09.812726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6kww5" event={"ID":"871ccb67-b9e3-4c8e-b3a9-bcf9e06dcfcb","Type":"ContainerStarted","Data":"97f3de012694ab61d722400ceee5900171d70592c2ff76388aca82cd8b09cb1d"} Jul 07 13:28:09.829458 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:09.829421 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6kww5" podStartSLOduration=5.198552687 podStartE2EDuration="23.829404148s" totalImagesPullingTime="18.630851461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:09.82890911 +0000 UTC m=+23.830150637" watchObservedRunningTime="2026-07-07 13:28:09.829404148 +0000 UTC m=+23.830645675" Jul 07 13:28:09.898605 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:09.898563 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 13:28:09.964402 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:09.964371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:28:09.965432 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:09.965403 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:28:10.471786 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:10.471574 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T13:28:09.898583737Z","UUID":"676634e6-49b5-4883-82dc-34e7f1a1d99a","Handler":null,"Name":"","Endpoint":""} Jul 07 13:28:10.475630 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:10.475565 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 13:28:10.475630 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:10.475611 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 13:28:10.584665 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:10.584634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:10.585148 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:10.584757 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:10.585148 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:10.585134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:10.585305 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:10.585252 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:10.818002 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:10.817963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" event={"ID":"6b7a7c00-45c0-4cc6-8ff2-000485810e8d","Type":"ContainerStarted","Data":"97d1422c9ce411b375b8684947271403179e72477463d39e5d104d8041498155"} Jul 07 13:28:11.584453 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:11.584421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:11.584684 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:11.584519 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:11.822524 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:11.822476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" event={"ID":"6b7a7c00-45c0-4cc6-8ff2-000485810e8d","Type":"ContainerStarted","Data":"f1d4944d5ae522c3088d01eb13753d47de04259d256ca1a710cd29e347422978"} Jul 07 13:28:11.825835 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:11.825810 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 13:28:11.825835 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:11.825826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" event={"ID":"e5e5fa67-7403-4ef6-920d-d3f15c658361","Type":"ContainerStarted","Data":"d2c445097c5b1c1fcf294d25af5f521dc3eaf81cf1035d2f854a12bfe3d7feb2"} Jul 07 13:28:12.123973 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:12.123792 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:28:12.124532 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:12.124508 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ns4bt" Jul 07 13:28:12.141288 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:12.141234 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48jrj" podStartSLOduration=5.071142798 podStartE2EDuration="26.141216538s" totalImagesPullingTime="21.07007374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:11.843122855 +0000 UTC m=+25.844364381" watchObservedRunningTime="2026-07-07 13:28:12.141216538 +0000 UTC m=+26.142458073" Jul 07 13:28:12.583685 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:12.583654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:12.583857 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:12.583666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:12.583857 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:12.583793 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:12.583969 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:12.583866 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:13.583764 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:13.583733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:13.584409 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:13.583840 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:13.831211 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:13.831126 2576 generic.go:356] "Generic (PLEG): container finished" podID="555fe057-e322-461c-9391-cfc0d2596f6f" containerID="43ed97bf95bc867fcb439ddcc1ab4e13cfcc64c7a701b005c966834dd7e6252e" exitCode=0 Jul 07 13:28:13.831339 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:13.831211 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t57fq" event={"ID":"555fe057-e322-461c-9391-cfc0d2596f6f","Type":"ContainerDied","Data":"43ed97bf95bc867fcb439ddcc1ab4e13cfcc64c7a701b005c966834dd7e6252e"} Jul 07 13:28:13.834917 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:13.834890 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" event={"ID":"e5e5fa67-7403-4ef6-920d-d3f15c658361","Type":"ContainerStarted","Data":"e4edbc2e00a58935e2250f7bf3398cccc693ed4161e1cd4a9f4fd7e210f8fd96"} Jul 07 13:28:14.584528 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:14.584366 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:14.584989 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:14.584366 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:14.584989 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:14.584638 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:14.584989 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:14.584699 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:14.837595 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:14.837533 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 13:28:14.837932 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:14.837914 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:28:14.838016 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:14.838005 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:28:14.853337 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:14.853311 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:28:14.853470 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:14.853405 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:28:14.887190 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:14.887117 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" podStartSLOduration=10.149155802 podStartE2EDuration="28.887103104s" totalImagesPullingTime="18.737947302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:13.948646333 +0000 UTC m=+27.949887870" watchObservedRunningTime="2026-07-07 13:28:14.887103104 +0000 UTC m=+28.888344647" Jul 07 13:28:15.583970 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:15.583948 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:15.584107 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:15.584035 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:15.631982 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:15.631959 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dcg6d"] Jul 07 13:28:15.641648 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:15.641625 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tq7cd"] Jul 07 13:28:15.641755 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:15.641718 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:15.641821 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:15.641800 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:15.642312 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:15.642281 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pgw8q"] Jul 07 13:28:15.642418 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:15.642390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:15.642503 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:15.642479 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:15.842025 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:15.841998 2576 generic.go:356] "Generic (PLEG): container finished" podID="555fe057-e322-461c-9391-cfc0d2596f6f" containerID="eb2dbd2bd77c6dac7098e42c6afd7e9aeeb2e1cc85f67aaf6e05f8113dbc6bf9" exitCode=0 Jul 07 13:28:15.842204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:15.842063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:15.842204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:15.842091 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t57fq" event={"ID":"555fe057-e322-461c-9391-cfc0d2596f6f","Type":"ContainerDied","Data":"eb2dbd2bd77c6dac7098e42c6afd7e9aeeb2e1cc85f67aaf6e05f8113dbc6bf9"} Jul 07 13:28:15.842323 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:15.842306 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 13:28:15.842371 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:15.842346 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:16.844715 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:16.844688 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 13:28:16.963212 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:16.963185 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:28:17.584472 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:17.584301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:17.584604 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:17.584346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:17.584604 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:17.584538 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:17.584685 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:17.584614 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:17.584685 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:17.584357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:17.584685 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:17.584682 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:17.849444 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:17.849413 2576 generic.go:356] "Generic (PLEG): container finished" podID="555fe057-e322-461c-9391-cfc0d2596f6f" containerID="0c2704b3f2337ac33ad4a358dda23cbb6024f9bf4314e0f9d0a701dbb146fc63" exitCode=0 Jul 07 13:28:17.849817 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:17.849508 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t57fq" event={"ID":"555fe057-e322-461c-9391-cfc0d2596f6f","Type":"ContainerDied","Data":"0c2704b3f2337ac33ad4a358dda23cbb6024f9bf4314e0f9d0a701dbb146fc63"} Jul 07 13:28:19.583745 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:19.583711 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:19.583745 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:19.583731 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:19.584148 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:19.583711 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:19.584148 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:19.583827 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgw8q" podUID="28b8a5de-4b31-4f32-b6b4-4e118987db04" Jul 07 13:28:19.584148 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:19.583874 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dcg6d" podUID="32f0bc68-c8ce-4756-8944-97f48df4e7cb" Jul 07 13:28:19.584148 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:19.583937 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7cd" podUID="bff31dac-b32a-4258-b236-2d5c0c58adc6" Jul 07 13:28:20.153444 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:20.153410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:20.153631 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:20.153551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:20.153631 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:20.153554 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 13:28:20.153749 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:20.153614 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:28:20.153749 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:20.153684 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret podName:32f0bc68-c8ce-4756-8944-97f48df4e7cb nodeName:}" failed. No retries permitted until 2026-07-07 13:28:52.15366422 +0000 UTC m=+66.154905727 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret") pod "global-pull-secret-syncer-dcg6d" (UID: "32f0bc68-c8ce-4756-8944-97f48df4e7cb") : object "kube-system"/"original-pull-secret" not registered Jul 07 13:28:20.153749 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:20.153735 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs podName:28b8a5de-4b31-4f32-b6b4-4e118987db04 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:52.153713783 +0000 UTC m=+66.154955311 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs") pod "network-metrics-daemon-pgw8q" (UID: "28b8a5de-4b31-4f32-b6b4-4e118987db04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 13:28:20.354924 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:20.354896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqx4\" (UniqueName: \"kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4\") pod \"network-check-target-tq7cd\" (UID: \"bff31dac-b32a-4258-b236-2d5c0c58adc6\") " pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:20.355092 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:20.355035 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 13:28:20.355092 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:20.355054 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 13:28:20.355092 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:20.355067 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2hqx4 for pod openshift-network-diagnostics/network-check-target-tq7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:28:20.355278 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:20.355123 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4 podName:bff31dac-b32a-4258-b236-2d5c0c58adc6 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:52.35510596 +0000 UTC m=+66.356347469 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hqx4" (UniqueName: "kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4") pod "network-check-target-tq7cd" (UID: "bff31dac-b32a-4258-b236-2d5c0c58adc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 13:28:21.328267 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.328230 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-236.ec2.internal" event="NodeReady" Jul 07 13:28:21.328718 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.328382 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 13:28:21.379474 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.379442 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xbmhz"] Jul 07 13:28:21.382416 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.382395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.385234 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.385208 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hzkh9"] Jul 07 13:28:21.385234 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.385226 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 13:28:21.385423 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.385339 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rgbgq\"" Jul 07 13:28:21.385493 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.385211 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 13:28:21.387500 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.387479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hzkh9" Jul 07 13:28:21.390245 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.390189 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 13:28:21.390375 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.390272 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 13:28:21.390482 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.390395 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 13:28:21.390482 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.390418 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-twdpw\"" Jul 07 13:28:21.392409 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.392391 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xbmhz"] Jul 07 13:28:21.396493 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.396471 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hzkh9"] Jul 07 13:28:21.462231 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.462195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.462379 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.462305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-tmp-dir\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.462379 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.462342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zl7q\" (UniqueName: \"kubernetes.io/projected/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-kube-api-access-7zl7q\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.462488 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.462387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b7cf248-7634-4651-b3d0-cb6f11b5bede-cert\") pod \"ingress-canary-hzkh9\" (UID: \"3b7cf248-7634-4651-b3d0-cb6f11b5bede\") " pod="openshift-ingress-canary/ingress-canary-hzkh9" Jul 07 13:28:21.462522 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.462484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhqlt\" (UniqueName: \"kubernetes.io/projected/3b7cf248-7634-4651-b3d0-cb6f11b5bede-kube-api-access-lhqlt\") pod \"ingress-canary-hzkh9\" (UID: \"3b7cf248-7634-4651-b3d0-cb6f11b5bede\") " pod="openshift-ingress-canary/ingress-canary-hzkh9" Jul 07 13:28:21.462570 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.462516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-config-volume\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.563461 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.563420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhqlt\" (UniqueName: \"kubernetes.io/projected/3b7cf248-7634-4651-b3d0-cb6f11b5bede-kube-api-access-lhqlt\") pod \"ingress-canary-hzkh9\" (UID: \"3b7cf248-7634-4651-b3d0-cb6f11b5bede\") " pod="openshift-ingress-canary/ingress-canary-hzkh9" Jul 07 13:28:21.563461 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.563468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-config-volume\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.563699 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.563534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.563699 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:21.563669 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 13:28:21.563803 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.563753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-tmp-dir\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.563803 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.563777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zl7q\" (UniqueName: \"kubernetes.io/projected/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-kube-api-access-7zl7q\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.563897 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.563838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b7cf248-7634-4651-b3d0-cb6f11b5bede-cert\") pod \"ingress-canary-hzkh9\" (UID: \"3b7cf248-7634-4651-b3d0-cb6f11b5bede\") " pod="openshift-ingress-canary/ingress-canary-hzkh9" Jul 07 13:28:21.563961 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:21.563943 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls podName:5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:22.063923213 +0000 UTC m=+36.065164730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls") pod "dns-default-xbmhz" (UID: "5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1") : secret "dns-default-metrics-tls" not found Jul 07 13:28:21.564102 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.564083 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-tmp-dir\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.564263 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.564235 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-config-volume\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.568048 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.568029 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b7cf248-7634-4651-b3d0-cb6f11b5bede-cert\") pod \"ingress-canary-hzkh9\" (UID: \"3b7cf248-7634-4651-b3d0-cb6f11b5bede\") " pod="openshift-ingress-canary/ingress-canary-hzkh9" Jul 07 13:28:21.574398 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.574375 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zl7q\" (UniqueName: \"kubernetes.io/projected/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-kube-api-access-7zl7q\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:21.574671 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.574644 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhqlt\" (UniqueName: \"kubernetes.io/projected/3b7cf248-7634-4651-b3d0-cb6f11b5bede-kube-api-access-lhqlt\") pod \"ingress-canary-hzkh9\" (UID: \"3b7cf248-7634-4651-b3d0-cb6f11b5bede\") " pod="openshift-ingress-canary/ingress-canary-hzkh9" Jul 07 13:28:21.584452 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.584390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:21.584543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.584485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:21.584594 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.584570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:21.587806 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.587783 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 13:28:21.587906 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.587814 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 13:28:21.588370 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.588215 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k6stv\"" Jul 07 13:28:21.588370 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.588228 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 13:28:21.588370 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.588220 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 13:28:21.588370 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.588319 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4z824\"" Jul 07 13:28:21.702003 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:21.701973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hzkh9" Jul 07 13:28:22.068028 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:22.067997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:22.068200 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:22.068155 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 13:28:22.068278 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:22.068231 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls podName:5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:23.068215654 +0000 UTC m=+37.069457169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls") pod "dns-default-xbmhz" (UID: "5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1") : secret "dns-default-metrics-tls" not found Jul 07 13:28:22.917648 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:22.917446 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hzkh9"] Jul 07 13:28:22.920953 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:22.920925 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7cf248_7634_4651_b3d0_cb6f11b5bede.slice/crio-49fab98b9386eda6191509a45c3c9d408a9ffa66984aaa14488e27f6a7f3a16f WatchSource:0}: Error finding container 49fab98b9386eda6191509a45c3c9d408a9ffa66984aaa14488e27f6a7f3a16f: Status 404 returned error can't find the container with id 49fab98b9386eda6191509a45c3c9d408a9ffa66984aaa14488e27f6a7f3a16f Jul 07 13:28:23.076601 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:23.076517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:23.076691 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:23.076627 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 13:28:23.076691 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:23.076685 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls podName:5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:25.076667399 +0000 UTC m=+39.077908907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls") pod "dns-default-xbmhz" (UID: "5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1") : secret "dns-default-metrics-tls" not found Jul 07 13:28:23.866875 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:23.866838 2576 generic.go:356] "Generic (PLEG): container finished" podID="555fe057-e322-461c-9391-cfc0d2596f6f" containerID="57de5ebbfc90051aa2e984ae80b6251bcfe4dcfe5cb627deac6787b21876271f" exitCode=0 Jul 07 13:28:23.867055 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:23.866920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t57fq" event={"ID":"555fe057-e322-461c-9391-cfc0d2596f6f","Type":"ContainerDied","Data":"57de5ebbfc90051aa2e984ae80b6251bcfe4dcfe5cb627deac6787b21876271f"} Jul 07 13:28:23.868209 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:23.868106 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hzkh9" event={"ID":"3b7cf248-7634-4651-b3d0-cb6f11b5bede","Type":"ContainerStarted","Data":"49fab98b9386eda6191509a45c3c9d408a9ffa66984aaa14488e27f6a7f3a16f"} Jul 07 13:28:24.874532 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:24.874498 2576 generic.go:356] "Generic (PLEG): container finished" podID="555fe057-e322-461c-9391-cfc0d2596f6f" containerID="0401c282ba2ce31bb30c53e8b57a99175113616ca53f68ef0252f03a42730a77" exitCode=0 Jul 07 13:28:24.874993 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:24.874570 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t57fq" event={"ID":"555fe057-e322-461c-9391-cfc0d2596f6f","Type":"ContainerDied","Data":"0401c282ba2ce31bb30c53e8b57a99175113616ca53f68ef0252f03a42730a77"} Jul 07 13:28:25.091773 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:25.091697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:25.091873 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:25.091811 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 13:28:25.091873 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:25.091873 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls podName:5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:29.091857024 +0000 UTC m=+43.093098538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls") pod "dns-default-xbmhz" (UID: "5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1") : secret "dns-default-metrics-tls" not found Jul 07 13:28:25.880071 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:25.879860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t57fq" event={"ID":"555fe057-e322-461c-9391-cfc0d2596f6f","Type":"ContainerStarted","Data":"80c45b37cc192e0c5ab65a590bf2bd70d5fa305501c3652f840a71d5039e525e"} Jul 07 13:28:25.881267 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:25.881243 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hzkh9" event={"ID":"3b7cf248-7634-4651-b3d0-cb6f11b5bede","Type":"ContainerStarted","Data":"b3338a0316fcc5307989bd5b21031aefca02fd130a564a56844d6b5e699d42d2"} Jul 07 13:28:25.912350 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:25.912309 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t57fq" podStartSLOduration=9.293586836 podStartE2EDuration="39.912296151s" totalImagesPullingTime="30.618709315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:25.91161446 +0000 UTC m=+39.912855990" watchObservedRunningTime="2026-07-07 13:28:25.912296151 +0000 UTC m=+39.913537709" Jul 07 13:28:25.933078 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:25.933043 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hzkh9" podStartSLOduration=2.496683997 podStartE2EDuration="4.933030992s" totalImagesPullingTime="2.436346995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:25.93261179 +0000 UTC m=+39.933853451" watchObservedRunningTime="2026-07-07 13:28:25.933030992 +0000 UTC m=+39.934272520" Jul 07 13:28:25.980598 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:25.980575 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rz9rq"] Jul 07 13:28:25.998776 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:25.998751 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rz9rq"] Jul 07 13:28:25.998885 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:25.998866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.006416 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.006397 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 13:28:26.006510 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.006420 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 13:28:26.006510 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.006399 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 13:28:26.007354 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.007336 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t5s5h\"" Jul 07 13:28:26.007696 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.007682 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 13:28:26.099736 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.099714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7trw\" (UniqueName: \"kubernetes.io/projected/4c603960-bc78-4176-bcce-153840f1df0b-kube-api-access-s7trw\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.099851 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.099794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c603960-bc78-4176-bcce-153840f1df0b-crio-socket\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.099851 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.099815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.099851 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.099832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c603960-bc78-4176-bcce-153840f1df0b-data-volume\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.099851 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.099847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.200085 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.200026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c603960-bc78-4176-bcce-153840f1df0b-crio-socket\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.200085 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.200054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.200085 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.200077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c603960-bc78-4176-bcce-153840f1df0b-data-volume\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.200319 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.200100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.200319 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.200137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7trw\" (UniqueName: \"kubernetes.io/projected/4c603960-bc78-4176-bcce-153840f1df0b-kube-api-access-s7trw\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.200319 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:26.200249 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 13:28:26.200319 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:26.200282 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 13:28:26.200472 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.200319 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c603960-bc78-4176-bcce-153840f1df0b-crio-socket\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.200472 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:26.200339 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm podName:4c603960-bc78-4176-bcce-153840f1df0b nodeName:}" failed. No retries permitted until 2026-07-07 13:28:26.700317177 +0000 UTC m=+40.701558935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rz9rq" (UID: "4c603960-bc78-4176-bcce-153840f1df0b") : configmap "kube-rbac-proxy" not found Jul 07 13:28:26.200472 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:26.200362 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls podName:4c603960-bc78-4176-bcce-153840f1df0b nodeName:}" failed. No retries permitted until 2026-07-07 13:28:26.700350555 +0000 UTC m=+40.701592065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rz9rq" (UID: "4c603960-bc78-4176-bcce-153840f1df0b") : secret "insights-runtime-extractor-tls" not found Jul 07 13:28:26.200472 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.200452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c603960-bc78-4176-bcce-153840f1df0b-data-volume\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.218229 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.218210 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7trw\" (UniqueName: \"kubernetes.io/projected/4c603960-bc78-4176-bcce-153840f1df0b-kube-api-access-s7trw\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.704224 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.704187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.704395 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:26.704230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:26.704395 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:26.704308 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 13:28:26.704395 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:26.704365 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm podName:4c603960-bc78-4176-bcce-153840f1df0b nodeName:}" failed. No retries permitted until 2026-07-07 13:28:27.70434768 +0000 UTC m=+41.705589186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rz9rq" (UID: "4c603960-bc78-4176-bcce-153840f1df0b") : configmap "kube-rbac-proxy" not found Jul 07 13:28:26.704571 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:26.704450 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 13:28:26.704571 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:26.704502 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls podName:4c603960-bc78-4176-bcce-153840f1df0b nodeName:}" failed. No retries permitted until 2026-07-07 13:28:27.704488226 +0000 UTC m=+41.705729731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rz9rq" (UID: "4c603960-bc78-4176-bcce-153840f1df0b") : secret "insights-runtime-extractor-tls" not found Jul 07 13:28:27.151000 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.150973 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc"] Jul 07 13:28:27.185449 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.185427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc"] Jul 07 13:28:27.185581 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.185520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc" Jul 07 13:28:27.188523 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.188506 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 13:28:27.188614 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.188564 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jdzvv\"" Jul 07 13:28:27.189848 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.189836 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 13:28:27.307458 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.307411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-884dt\" (UniqueName: \"kubernetes.io/projected/f2d1f3d3-c447-4f28-ab19-c9ae8554f58b-kube-api-access-884dt\") pod \"migrator-d57d58cc9-tsxzc\" (UID: \"f2d1f3d3-c447-4f28-ab19-c9ae8554f58b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc" Jul 07 13:28:27.408322 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.408253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-884dt\" (UniqueName: \"kubernetes.io/projected/f2d1f3d3-c447-4f28-ab19-c9ae8554f58b-kube-api-access-884dt\") pod \"migrator-d57d58cc9-tsxzc\" (UID: \"f2d1f3d3-c447-4f28-ab19-c9ae8554f58b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc" Jul 07 13:28:27.416401 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.416373 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-884dt\" (UniqueName: \"kubernetes.io/projected/f2d1f3d3-c447-4f28-ab19-c9ae8554f58b-kube-api-access-884dt\") pod \"migrator-d57d58cc9-tsxzc\" (UID: \"f2d1f3d3-c447-4f28-ab19-c9ae8554f58b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc" Jul 07 13:28:27.494447 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.494419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc" Jul 07 13:28:27.626131 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.626103 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc"] Jul 07 13:28:27.629676 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:27.629649 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d1f3d3_c447_4f28_ab19_c9ae8554f58b.slice/crio-a85f0f443baaa528c6d2edb52d0e018eb74d1a35785a42850bce84b0993cc943 WatchSource:0}: Error finding container a85f0f443baaa528c6d2edb52d0e018eb74d1a35785a42850bce84b0993cc943: Status 404 returned error can't find the container with id a85f0f443baaa528c6d2edb52d0e018eb74d1a35785a42850bce84b0993cc943 Jul 07 13:28:27.689122 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.689063 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw"] Jul 07 13:28:27.711399 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.711376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:27.711521 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.711415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:27.711568 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:27.711528 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 13:28:27.711568 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:27.711547 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 13:28:27.711641 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:27.711583 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls podName:4c603960-bc78-4176-bcce-153840f1df0b nodeName:}" failed. No retries permitted until 2026-07-07 13:28:29.711568718 +0000 UTC m=+43.712810227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rz9rq" (UID: "4c603960-bc78-4176-bcce-153840f1df0b") : secret "insights-runtime-extractor-tls" not found Jul 07 13:28:27.711641 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:27.711597 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm podName:4c603960-bc78-4176-bcce-153840f1df0b nodeName:}" failed. No retries permitted until 2026-07-07 13:28:29.711590607 +0000 UTC m=+43.712832112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rz9rq" (UID: "4c603960-bc78-4176-bcce-153840f1df0b") : configmap "kube-rbac-proxy" not found Jul 07 13:28:27.721244 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.721226 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw"] Jul 07 13:28:27.721321 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.721311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:27.724160 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.724144 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 07 13:28:27.724251 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.724157 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wcpcl\"" Jul 07 13:28:27.724352 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.724337 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 07 13:28:27.812527 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.812504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f2851ac-4bb5-4cb4-957b-146cec3af69c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:27.812634 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.812532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:27.886765 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.886740 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc" event={"ID":"f2d1f3d3-c447-4f28-ab19-c9ae8554f58b","Type":"ContainerStarted","Data":"a85f0f443baaa528c6d2edb52d0e018eb74d1a35785a42850bce84b0993cc943"} Jul 07 13:28:27.913478 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.913454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f2851ac-4bb5-4cb4-957b-146cec3af69c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:27.913578 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.913489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:27.913631 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:27.913614 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 13:28:27.913685 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:27.913679 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert podName:0f2851ac-4bb5-4cb4-957b-146cec3af69c nodeName:}" failed. No retries permitted until 2026-07-07 13:28:28.413660082 +0000 UTC m=+42.414901587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qd2sw" (UID: "0f2851ac-4bb5-4cb4-957b-146cec3af69c") : secret "networking-console-plugin-cert" not found Jul 07 13:28:27.914096 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:27.914076 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f2851ac-4bb5-4cb4-957b-146cec3af69c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:28.155196 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:28.153562 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qhhwm_0165e63b-34ba-494b-9a7b-31eb2e652b54/dns-node-resolver/0.log" Jul 07 13:28:28.417273 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:28.417185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:28.417436 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:28.417353 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 13:28:28.417436 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:28.417422 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert podName:0f2851ac-4bb5-4cb4-957b-146cec3af69c nodeName:}" failed. No retries permitted until 2026-07-07 13:28:29.417404975 +0000 UTC m=+43.418646493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qd2sw" (UID: "0f2851ac-4bb5-4cb4-957b-146cec3af69c") : secret "networking-console-plugin-cert" not found Jul 07 13:28:28.953626 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:28.953586 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wcqgp_17a9a9ab-5b3b-4805-b971-3be2d30b46ec/node-ca/0.log" Jul 07 13:28:29.121700 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:29.121665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:29.121833 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:29.121788 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 13:28:29.121885 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:29.121849 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls podName:5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:37.121836278 +0000 UTC m=+51.123077782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls") pod "dns-default-xbmhz" (UID: "5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1") : secret "dns-default-metrics-tls" not found Jul 07 13:28:29.423886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:29.423856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:29.424157 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:29.423970 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 13:28:29.424157 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:29.424035 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert podName:0f2851ac-4bb5-4cb4-957b-146cec3af69c nodeName:}" failed. No retries permitted until 2026-07-07 13:28:31.424018223 +0000 UTC m=+45.425259729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qd2sw" (UID: "0f2851ac-4bb5-4cb4-957b-146cec3af69c") : secret "networking-console-plugin-cert" not found Jul 07 13:28:29.724971 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:29.724945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:29.725083 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:29.724981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:29.725129 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:29.725112 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 13:28:29.725198 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:29.725188 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls podName:4c603960-bc78-4176-bcce-153840f1df0b nodeName:}" failed. No retries permitted until 2026-07-07 13:28:33.725154695 +0000 UTC m=+47.726396203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rz9rq" (UID: "4c603960-bc78-4176-bcce-153840f1df0b") : secret "insights-runtime-extractor-tls" not found Jul 07 13:28:29.725441 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:29.725426 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c603960-bc78-4176-bcce-153840f1df0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:29.752315 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:29.752295 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hzkh9_3b7cf248-7634-4651-b3d0-cb6f11b5bede/serve-healthcheck-canary/0.log" Jul 07 13:28:29.893852 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:29.893826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc" event={"ID":"f2d1f3d3-c447-4f28-ab19-c9ae8554f58b","Type":"ContainerStarted","Data":"1e757a9fb76ae9df00034aef199aa39e26744d68bdb6c2bf82dc21bf285d968a"} Jul 07 13:28:29.893951 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:29.893871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc" event={"ID":"f2d1f3d3-c447-4f28-ab19-c9ae8554f58b","Type":"ContainerStarted","Data":"615008b853c457c98e4dc5247c81a3b0b77f9add0f2b580759952818f6a9adbe"} Jul 07 13:28:29.925934 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:29.925891 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-tsxzc" podStartSLOduration=1.1513313840000001 podStartE2EDuration="2.925876814s" totalImagesPullingTime="1.77454543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:29.925499202 +0000 UTC m=+43.926740730" watchObservedRunningTime="2026-07-07 13:28:29.925876814 +0000 UTC m=+43.927118386" Jul 07 13:28:31.436585 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:31.436551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:31.436978 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:31.436662 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 13:28:31.436978 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:31.436721 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert podName:0f2851ac-4bb5-4cb4-957b-146cec3af69c nodeName:}" failed. No retries permitted until 2026-07-07 13:28:35.436704966 +0000 UTC m=+49.437946492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qd2sw" (UID: "0f2851ac-4bb5-4cb4-957b-146cec3af69c") : secret "networking-console-plugin-cert" not found Jul 07 13:28:33.753756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:33.753714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:33.754213 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:33.753875 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 13:28:33.754213 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:33.753940 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls podName:4c603960-bc78-4176-bcce-153840f1df0b nodeName:}" failed. No retries permitted until 2026-07-07 13:28:41.753923161 +0000 UTC m=+55.755164683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rz9rq" (UID: "4c603960-bc78-4176-bcce-153840f1df0b") : secret "insights-runtime-extractor-tls" not found Jul 07 13:28:35.466449 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:35.466413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:35.466817 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:35.466556 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 13:28:35.466817 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:35.466618 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert podName:0f2851ac-4bb5-4cb4-957b-146cec3af69c nodeName:}" failed. No retries permitted until 2026-07-07 13:28:43.466602617 +0000 UTC m=+57.467844137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qd2sw" (UID: "0f2851ac-4bb5-4cb4-957b-146cec3af69c") : secret "networking-console-plugin-cert" not found Jul 07 13:28:37.180723 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:37.180689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:37.181086 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:37.180834 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 13:28:37.181086 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:28:37.180897 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls podName:5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1 nodeName:}" failed. No retries permitted until 2026-07-07 13:28:53.180883119 +0000 UTC m=+67.182124624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls") pod "dns-default-xbmhz" (UID: "5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1") : secret "dns-default-metrics-tls" not found Jul 07 13:28:41.811822 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:41.811778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:41.814147 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:41.814118 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c603960-bc78-4176-bcce-153840f1df0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz9rq\" (UID: \"4c603960-bc78-4176-bcce-153840f1df0b\") " pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:41.908588 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:41.908555 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rz9rq" Jul 07 13:28:42.047448 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:42.047418 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rz9rq"] Jul 07 13:28:42.051139 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:42.051115 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c603960_bc78_4176_bcce_153840f1df0b.slice/crio-741bf583fb0e414f4a39d91adb0a2be3a4481858d44381b06c6cfc8aa99c86a3 WatchSource:0}: Error finding container 741bf583fb0e414f4a39d91adb0a2be3a4481858d44381b06c6cfc8aa99c86a3: Status 404 returned error can't find the container with id 741bf583fb0e414f4a39d91adb0a2be3a4481858d44381b06c6cfc8aa99c86a3 Jul 07 13:28:42.929153 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:42.929112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rz9rq" event={"ID":"4c603960-bc78-4176-bcce-153840f1df0b","Type":"ContainerStarted","Data":"741bf583fb0e414f4a39d91adb0a2be3a4481858d44381b06c6cfc8aa99c86a3"} Jul 07 13:28:43.526025 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:43.525992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:43.528593 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:43.528563 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f2851ac-4bb5-4cb4-957b-146cec3af69c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qd2sw\" (UID: \"0f2851ac-4bb5-4cb4-957b-146cec3af69c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:43.630309 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:43.630275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" Jul 07 13:28:45.724626 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:45.724598 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw"] Jul 07 13:28:45.728956 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:45.728931 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2851ac_4bb5_4cb4_957b_146cec3af69c.slice/crio-5ea4df876eda26346efa86a448f6fe3586e8189463abac209cc67e1c7102f484 WatchSource:0}: Error finding container 5ea4df876eda26346efa86a448f6fe3586e8189463abac209cc67e1c7102f484: Status 404 returned error can't find the container with id 5ea4df876eda26346efa86a448f6fe3586e8189463abac209cc67e1c7102f484 Jul 07 13:28:45.936521 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:45.936444 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" event={"ID":"0f2851ac-4bb5-4cb4-957b-146cec3af69c","Type":"ContainerStarted","Data":"5ea4df876eda26346efa86a448f6fe3586e8189463abac209cc67e1c7102f484"} Jul 07 13:28:45.937776 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:45.937754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rz9rq" event={"ID":"4c603960-bc78-4176-bcce-153840f1df0b","Type":"ContainerStarted","Data":"fe783a9c92a4e9acb221af8b349bb85ea8de9fdbd40123c370e8ccc14cf87cc7"} Jul 07 13:28:46.942025 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:46.941984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rz9rq" event={"ID":"4c603960-bc78-4176-bcce-153840f1df0b","Type":"ContainerStarted","Data":"3c697ae04323f12054fca49fb0aefc26d3e7567547273f317d7db4ab4639d7b3"} Jul 07 13:28:47.871186 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:47.871145 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnr4d" Jul 07 13:28:47.947932 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:47.947897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" event={"ID":"0f2851ac-4bb5-4cb4-957b-146cec3af69c","Type":"ContainerStarted","Data":"3393dcd7e1c7d4d2606f8111aafbc3686c4b6a921428091dc489c98cbd9ace02"} Jul 07 13:28:47.967554 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:47.967507 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-qd2sw" podStartSLOduration=19.470726136 podStartE2EDuration="20.967493597s" totalImagesPullingTime="1.496767461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:47.966603933 +0000 UTC m=+61.967845462" watchObservedRunningTime="2026-07-07 13:28:47.967493597 +0000 UTC m=+61.968735124" Jul 07 13:28:49.956001 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:49.955955 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rz9rq" event={"ID":"4c603960-bc78-4176-bcce-153840f1df0b","Type":"ContainerStarted","Data":"c83916c3a0bd2ad5e0e83b911061b9383214d30df1e85337acdfe0646668fad2"} Jul 07 13:28:49.975509 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:49.975462 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rz9rq" podStartSLOduration=17.962467658 podStartE2EDuration="24.975447746s" totalImagesPullingTime="7.012980088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:28:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:49.974178274 +0000 UTC m=+63.975419797" watchObservedRunningTime="2026-07-07 13:28:49.975447746 +0000 UTC m=+63.976689263" Jul 07 13:28:50.924241 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.924206 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r"] Jul 07 13:28:50.926273 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.926257 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:50.930554 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.930499 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 13:28:50.930882 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.930847 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 13:28:50.931487 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.931108 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 13:28:50.931487 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.931142 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 13:28:50.931487 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.930962 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 13:28:50.931487 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.930962 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 13:28:50.931487 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.931408 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 13:28:50.941266 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.941246 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r"] Jul 07 13:28:50.984236 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.984214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:50.984578 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.984276 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6k79\" (UniqueName: \"kubernetes.io/projected/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-kube-api-access-g6k79\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:50.984578 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.984365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:50.984578 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.984422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-ca\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:50.984578 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.984481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-hub\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:50.984578 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:50.984511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.056584 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.056557 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-zzfs7"] Jul 07 13:28:51.058576 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.058562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-zzfs7" Jul 07 13:28:51.061359 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.061337 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 13:28:51.061467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.061389 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 13:28:51.061528 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.061482 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8bvgp\"" Jul 07 13:28:51.077609 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.077589 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-zzfs7"] Jul 07 13:28:51.085149 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.085132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65cfq\" (UniqueName: \"kubernetes.io/projected/ac8db3f2-4183-4674-95b3-b7ea6ab06b58-kube-api-access-65cfq\") pod \"downloads-685bfb5f64-zzfs7\" (UID: \"ac8db3f2-4183-4674-95b3-b7ea6ab06b58\") " pod="openshift-console/downloads-685bfb5f64-zzfs7" Jul 07 13:28:51.085214 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.085179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.085250 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.085231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-ca\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.085281 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.085258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-hub\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.085323 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.085279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.085372 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.085321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.085372 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.085355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6k79\" (UniqueName: \"kubernetes.io/projected/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-kube-api-access-g6k79\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.085743 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.085720 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.087671 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.087645 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-hub\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.087773 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.087651 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.087773 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.087656 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-ca\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.087773 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.087762 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.102834 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.102814 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6k79\" (UniqueName: \"kubernetes.io/projected/ef7f4a07-2d3f-4331-a2e0-eb9df830674a-kube-api-access-g6k79\") pod \"cluster-proxy-proxy-agent-76488fc748-2gw6r\" (UID: \"ef7f4a07-2d3f-4331-a2e0-eb9df830674a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.160896 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.160873 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84dcfb865b-s2qps"] Jul 07 13:28:51.162851 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.162839 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.166477 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.166461 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 13:28:51.166663 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.166648 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 13:28:51.168952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.168931 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 13:28:51.170280 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.170264 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mmkl7\"" Jul 07 13:28:51.176190 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.176135 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 13:28:51.186363 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.186341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2bd6ca3-d88a-456f-913c-0ba058905a66-registry-certificates\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.186454 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.186379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2bd6ca3-d88a-456f-913c-0ba058905a66-installation-pull-secrets\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.186454 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.186412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2bd6ca3-d88a-456f-913c-0ba058905a66-ca-trust-extracted\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.186542 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.186485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2bd6ca3-d88a-456f-913c-0ba058905a66-bound-sa-token\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.186542 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.186538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkp9r\" (UniqueName: \"kubernetes.io/projected/b2bd6ca3-d88a-456f-913c-0ba058905a66-kube-api-access-qkp9r\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.186623 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.186560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2bd6ca3-d88a-456f-913c-0ba058905a66-image-registry-private-configuration\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.186623 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.186583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2bd6ca3-d88a-456f-913c-0ba058905a66-registry-tls\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.186723 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.186630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2bd6ca3-d88a-456f-913c-0ba058905a66-trusted-ca\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.186723 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.186700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65cfq\" (UniqueName: \"kubernetes.io/projected/ac8db3f2-4183-4674-95b3-b7ea6ab06b58-kube-api-access-65cfq\") pod \"downloads-685bfb5f64-zzfs7\" (UID: \"ac8db3f2-4183-4674-95b3-b7ea6ab06b58\") " pod="openshift-console/downloads-685bfb5f64-zzfs7" Jul 07 13:28:51.187246 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.187227 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84dcfb865b-s2qps"] Jul 07 13:28:51.200291 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.200271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65cfq\" (UniqueName: \"kubernetes.io/projected/ac8db3f2-4183-4674-95b3-b7ea6ab06b58-kube-api-access-65cfq\") pod \"downloads-685bfb5f64-zzfs7\" (UID: \"ac8db3f2-4183-4674-95b3-b7ea6ab06b58\") " pod="openshift-console/downloads-685bfb5f64-zzfs7" Jul 07 13:28:51.254793 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.254772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" Jul 07 13:28:51.287233 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.287207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkp9r\" (UniqueName: \"kubernetes.io/projected/b2bd6ca3-d88a-456f-913c-0ba058905a66-kube-api-access-qkp9r\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.287354 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.287249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2bd6ca3-d88a-456f-913c-0ba058905a66-image-registry-private-configuration\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.287354 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.287282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2bd6ca3-d88a-456f-913c-0ba058905a66-registry-tls\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.287354 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.287327 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2bd6ca3-d88a-456f-913c-0ba058905a66-trusted-ca\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.287512 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.287382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2bd6ca3-d88a-456f-913c-0ba058905a66-registry-certificates\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.287512 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.287413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2bd6ca3-d88a-456f-913c-0ba058905a66-installation-pull-secrets\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.287512 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.287447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2bd6ca3-d88a-456f-913c-0ba058905a66-ca-trust-extracted\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.287512 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.287479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2bd6ca3-d88a-456f-913c-0ba058905a66-bound-sa-token\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.288042 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.288019 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2bd6ca3-d88a-456f-913c-0ba058905a66-ca-trust-extracted\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.288484 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.288456 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2bd6ca3-d88a-456f-913c-0ba058905a66-registry-certificates\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.288789 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.288764 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2bd6ca3-d88a-456f-913c-0ba058905a66-trusted-ca\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.290043 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.289988 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2bd6ca3-d88a-456f-913c-0ba058905a66-installation-pull-secrets\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.290207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.290156 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2bd6ca3-d88a-456f-913c-0ba058905a66-image-registry-private-configuration\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.290616 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.290591 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2bd6ca3-d88a-456f-913c-0ba058905a66-registry-tls\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.305024 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.304996 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkp9r\" (UniqueName: \"kubernetes.io/projected/b2bd6ca3-d88a-456f-913c-0ba058905a66-kube-api-access-qkp9r\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.310326 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.310305 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2bd6ca3-d88a-456f-913c-0ba058905a66-bound-sa-token\") pod \"image-registry-84dcfb865b-s2qps\" (UID: \"b2bd6ca3-d88a-456f-913c-0ba058905a66\") " pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.366836 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.366805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-zzfs7" Jul 07 13:28:51.381423 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.381401 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r"] Jul 07 13:28:51.384451 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:51.384429 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef7f4a07_2d3f_4331_a2e0_eb9df830674a.slice/crio-914d54f9452ceb8a156165383f4f6e560f3e3b2fd6cc0d2dc8e0a940aa1217f6 WatchSource:0}: Error finding container 914d54f9452ceb8a156165383f4f6e560f3e3b2fd6cc0d2dc8e0a940aa1217f6: Status 404 returned error can't find the container with id 914d54f9452ceb8a156165383f4f6e560f3e3b2fd6cc0d2dc8e0a940aa1217f6 Jul 07 13:28:51.470767 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.470744 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.487104 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.487083 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-zzfs7"] Jul 07 13:28:51.489015 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:51.488992 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8db3f2_4183_4674_95b3_b7ea6ab06b58.slice/crio-72ae530f74027d627fda6c36d522e3cdaedd54b3ef6b6a7d5c4c4aa8c097c222 WatchSource:0}: Error finding container 72ae530f74027d627fda6c36d522e3cdaedd54b3ef6b6a7d5c4c4aa8c097c222: Status 404 returned error can't find the container with id 72ae530f74027d627fda6c36d522e3cdaedd54b3ef6b6a7d5c4c4aa8c097c222 Jul 07 13:28:51.592275 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.592253 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84dcfb865b-s2qps"] Jul 07 13:28:51.593897 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:51.593870 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bd6ca3_d88a_456f_913c_0ba058905a66.slice/crio-18cd324344c7b4ef12af6d50fddcbfc471ed91e97e9cf5caba45a35bd3b3d822 WatchSource:0}: Error finding container 18cd324344c7b4ef12af6d50fddcbfc471ed91e97e9cf5caba45a35bd3b3d822: Status 404 returned error can't find the container with id 18cd324344c7b4ef12af6d50fddcbfc471ed91e97e9cf5caba45a35bd3b3d822 Jul 07 13:28:51.967873 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.967836 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" event={"ID":"ef7f4a07-2d3f-4331-a2e0-eb9df830674a","Type":"ContainerStarted","Data":"914d54f9452ceb8a156165383f4f6e560f3e3b2fd6cc0d2dc8e0a940aa1217f6"} Jul 07 13:28:51.968980 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.968940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-zzfs7" event={"ID":"ac8db3f2-4183-4674-95b3-b7ea6ab06b58","Type":"ContainerStarted","Data":"72ae530f74027d627fda6c36d522e3cdaedd54b3ef6b6a7d5c4c4aa8c097c222"} Jul 07 13:28:51.970371 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.970347 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" event={"ID":"b2bd6ca3-d88a-456f-913c-0ba058905a66","Type":"ContainerStarted","Data":"6fe15d03e88340e2c2ea9c66305f9fabc660f4628eb4a2c1f6924b33fc1cb269"} Jul 07 13:28:51.970494 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.970378 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" event={"ID":"b2bd6ca3-d88a-456f-913c-0ba058905a66","Type":"ContainerStarted","Data":"18cd324344c7b4ef12af6d50fddcbfc471ed91e97e9cf5caba45a35bd3b3d822"} Jul 07 13:28:51.970494 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.970476 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:28:51.991253 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:51.991207 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" podStartSLOduration=0.991191373 podStartE2EDuration="991.191373ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:28:51.990455389 +0000 UTC m=+65.991696920" watchObservedRunningTime="2026-07-07 13:28:51.991191373 +0000 UTC m=+65.992432900" Jul 07 13:28:52.193902 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.193867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:52.194077 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.194034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:52.197219 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.197196 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 13:28:52.197319 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.197225 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 13:28:52.207385 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.207361 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32f0bc68-c8ce-4756-8944-97f48df4e7cb-original-pull-secret\") pod \"global-pull-secret-syncer-dcg6d\" (UID: \"32f0bc68-c8ce-4756-8944-97f48df4e7cb\") " pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:52.207473 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.207437 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b8a5de-4b31-4f32-b6b4-4e118987db04-metrics-certs\") pod \"network-metrics-daemon-pgw8q\" (UID: \"28b8a5de-4b31-4f32-b6b4-4e118987db04\") " pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:52.210672 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.210651 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k6stv\"" Jul 07 13:28:52.218959 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.218907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgw8q" Jul 07 13:28:52.352114 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.352073 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pgw8q"] Jul 07 13:28:52.355595 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:52.355566 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b8a5de_4b31_4f32_b6b4_4e118987db04.slice/crio-1eb2a4c01ad35ae28c1e5b4b61bb3da93e0f0a95fce76ee33aa02fa1dd7c1925 WatchSource:0}: Error finding container 1eb2a4c01ad35ae28c1e5b4b61bb3da93e0f0a95fce76ee33aa02fa1dd7c1925: Status 404 returned error can't find the container with id 1eb2a4c01ad35ae28c1e5b4b61bb3da93e0f0a95fce76ee33aa02fa1dd7c1925 Jul 07 13:28:52.395707 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.395466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqx4\" (UniqueName: \"kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4\") pod \"network-check-target-tq7cd\" (UID: \"bff31dac-b32a-4258-b236-2d5c0c58adc6\") " pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:52.398543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.398521 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 13:28:52.409841 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.409821 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 13:28:52.421114 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.421089 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqx4\" (UniqueName: \"kubernetes.io/projected/bff31dac-b32a-4258-b236-2d5c0c58adc6-kube-api-access-2hqx4\") pod \"network-check-target-tq7cd\" (UID: \"bff31dac-b32a-4258-b236-2d5c0c58adc6\") " pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:52.498973 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.498946 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4z824\"" Jul 07 13:28:52.502540 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.502518 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dcg6d" Jul 07 13:28:52.506888 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.506869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:28:52.680394 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.680370 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dcg6d"] Jul 07 13:28:52.682963 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:52.682926 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f0bc68_c8ce_4756_8944_97f48df4e7cb.slice/crio-bb6cce0f89178688a245e9f65deef3f11c497eeeb1c58475fc7454943afc97a3 WatchSource:0}: Error finding container bb6cce0f89178688a245e9f65deef3f11c497eeeb1c58475fc7454943afc97a3: Status 404 returned error can't find the container with id bb6cce0f89178688a245e9f65deef3f11c497eeeb1c58475fc7454943afc97a3 Jul 07 13:28:52.709550 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.709515 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tq7cd"] Jul 07 13:28:52.714115 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:52.714081 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff31dac_b32a_4258_b236_2d5c0c58adc6.slice/crio-29903b2ce505302838b3c0bea62217bf748f119689461075ddfb544ea327a962 WatchSource:0}: Error finding container 29903b2ce505302838b3c0bea62217bf748f119689461075ddfb544ea327a962: Status 404 returned error can't find the container with id 29903b2ce505302838b3c0bea62217bf748f119689461075ddfb544ea327a962 Jul 07 13:28:52.978451 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.978416 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dcg6d" event={"ID":"32f0bc68-c8ce-4756-8944-97f48df4e7cb","Type":"ContainerStarted","Data":"bb6cce0f89178688a245e9f65deef3f11c497eeeb1c58475fc7454943afc97a3"} Jul 07 13:28:52.979922 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.979894 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pgw8q" event={"ID":"28b8a5de-4b31-4f32-b6b4-4e118987db04","Type":"ContainerStarted","Data":"1eb2a4c01ad35ae28c1e5b4b61bb3da93e0f0a95fce76ee33aa02fa1dd7c1925"} Jul 07 13:28:52.982999 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:52.982937 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tq7cd" event={"ID":"bff31dac-b32a-4258-b236-2d5c0c58adc6","Type":"ContainerStarted","Data":"29903b2ce505302838b3c0bea62217bf748f119689461075ddfb544ea327a962"} Jul 07 13:28:53.203004 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:53.202454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:53.209672 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:53.209616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1-metrics-tls\") pod \"dns-default-xbmhz\" (UID: \"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1\") " pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:53.498925 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:53.498685 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rgbgq\"" Jul 07 13:28:53.507051 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:53.506782 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xbmhz" Jul 07 13:28:59.757835 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:28:59.756988 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xbmhz"] Jul 07 13:28:59.768881 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:28:59.768854 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8603c4_97f0_499b_8cc4_ae1ea0b71ea1.slice/crio-ebc6ceb30172e695cac53e8d4c58ecdd575e5409e101027775a3b5c63c11cd93 WatchSource:0}: Error finding container ebc6ceb30172e695cac53e8d4c58ecdd575e5409e101027775a3b5c63c11cd93: Status 404 returned error can't find the container with id ebc6ceb30172e695cac53e8d4c58ecdd575e5409e101027775a3b5c63c11cd93 Jul 07 13:29:00.019352 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.019310 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tq7cd" event={"ID":"bff31dac-b32a-4258-b236-2d5c0c58adc6","Type":"ContainerStarted","Data":"279369b55e6716ebcba408392354c6fde82c0707ca429b06c55afeb921e687d0"} Jul 07 13:29:00.019541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.019419 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:29:00.021022 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.020980 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dcg6d" event={"ID":"32f0bc68-c8ce-4756-8944-97f48df4e7cb","Type":"ContainerStarted","Data":"dd069d318a28827af0a65acda3ab26328831814c229b22a0eb003b20ae0dafd9"} Jul 07 13:29:00.022284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.022249 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbmhz" event={"ID":"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1","Type":"ContainerStarted","Data":"ebc6ceb30172e695cac53e8d4c58ecdd575e5409e101027775a3b5c63c11cd93"} Jul 07 13:29:00.024527 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.024505 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" event={"ID":"ef7f4a07-2d3f-4331-a2e0-eb9df830674a","Type":"ContainerStarted","Data":"faa09d9144ea81db60964a4252268e6f3ac877aaa66d617568cd1fb375d20ad4"} Jul 07 13:29:00.024643 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.024535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" event={"ID":"ef7f4a07-2d3f-4331-a2e0-eb9df830674a","Type":"ContainerStarted","Data":"63000c2d1f170b22247b5e461036c8cb847a245522a409a31cc1fff109eb3258"} Jul 07 13:29:00.024643 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.024551 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" event={"ID":"ef7f4a07-2d3f-4331-a2e0-eb9df830674a","Type":"ContainerStarted","Data":"90e722acc195f1fc4600d0de3dae1fe0f6689875276d00928fd53446fd46f513"} Jul 07 13:29:00.026412 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.026388 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pgw8q" event={"ID":"28b8a5de-4b31-4f32-b6b4-4e118987db04","Type":"ContainerStarted","Data":"df58fb483cb39b244d33bb10efba172386ebda5cac22e2275d885f59444b7cd4"} Jul 07 13:29:00.026512 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.026434 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pgw8q" event={"ID":"28b8a5de-4b31-4f32-b6b4-4e118987db04","Type":"ContainerStarted","Data":"50863a1979b250d64d98261cd4da85ad91042f7c852c7220714c9bee1378f573"} Jul 07 13:29:00.038149 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.038111 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tq7cd" podStartSLOduration=67.141918699 podStartE2EDuration="1m14.038096748s" totalImagesPullingTime="6.896178049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:29:00.037988785 +0000 UTC m=+74.039230313" watchObservedRunningTime="2026-07-07 13:29:00.038096748 +0000 UTC m=+74.039338276" Jul 07 13:29:00.054318 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.054272 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dcg6d" podStartSLOduration=67.131253274 podStartE2EDuration="1m14.05426067s" totalImagesPullingTime="6.923007396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:29:00.054037754 +0000 UTC m=+74.055279282" watchObservedRunningTime="2026-07-07 13:29:00.05426067 +0000 UTC m=+74.055502197" Jul 07 13:29:00.072706 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.072666 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pgw8q" podStartSLOduration=66.847918654 podStartE2EDuration="1m14.072655439s" totalImagesPullingTime="7.224736785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:29:00.072277451 +0000 UTC m=+74.073518978" watchObservedRunningTime="2026-07-07 13:29:00.072655439 +0000 UTC m=+74.073896965" Jul 07 13:29:00.095095 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:00.095028 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76488fc748-2gw6r" podStartSLOduration=1.891073724 podStartE2EDuration="10.095016377s" totalImagesPullingTime="8.203942653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:29:00.094282716 +0000 UTC m=+74.095524244" watchObservedRunningTime="2026-07-07 13:29:00.095016377 +0000 UTC m=+74.096257905" Jul 07 13:29:02.036596 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:02.036503 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbmhz" event={"ID":"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1","Type":"ContainerStarted","Data":"7e306527ed0c43dc3fa86611d4dd7e41bd38e6e173331d7647f887bc619fb45e"} Jul 07 13:29:02.036596 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:02.036562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbmhz" event={"ID":"5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1","Type":"ContainerStarted","Data":"d0238ecfc1833a5ed2ea666562c5bb6f143454c92a7b4dffcd3e49795ae1dc96"} Jul 07 13:29:02.037082 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:02.036635 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xbmhz" Jul 07 13:29:02.059578 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:02.059509 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xbmhz" podStartSLOduration=39.090682796 podStartE2EDuration="41.059492203s" totalImagesPullingTime="1.968809407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:29:02.057882954 +0000 UTC m=+76.059124482" watchObservedRunningTime="2026-07-07 13:29:02.059492203 +0000 UTC m=+76.060733729" Jul 07 13:29:06.904970 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:06.904895 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qkq9g"] Jul 07 13:29:06.919961 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:06.919924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:06.923401 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:06.923372 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 13:29:06.923720 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:06.923697 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 13:29:06.923870 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:06.923762 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 13:29:06.924136 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:06.924119 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 13:29:06.924953 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:06.924937 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 13:29:06.925047 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:06.924917 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6z2kr\"" Jul 07 13:29:06.933205 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:06.933185 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 13:29:07.006016 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.005974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-tls\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.006229 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.006011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-wtmp\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.006229 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.006134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f41a94cc-0686-41ca-9666-2223a9736bd7-root\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.006229 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.006194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-accelerators-collector-config\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.006229 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.006225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f41a94cc-0686-41ca-9666-2223a9736bd7-metrics-client-ca\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.006452 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.006253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.006452 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.006279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-textfile\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.006452 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.006381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f41a94cc-0686-41ca-9666-2223a9736bd7-sys\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.006452 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.006425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nw2l\" (UniqueName: \"kubernetes.io/projected/f41a94cc-0686-41ca-9666-2223a9736bd7-kube-api-access-6nw2l\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.107549 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f41a94cc-0686-41ca-9666-2223a9736bd7-root\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.107549 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-accelerators-collector-config\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.107792 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f41a94cc-0686-41ca-9666-2223a9736bd7-metrics-client-ca\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.107792 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.107792 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-textfile\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.107792 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107641 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f41a94cc-0686-41ca-9666-2223a9736bd7-root\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.107792 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f41a94cc-0686-41ca-9666-2223a9736bd7-sys\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.107792 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nw2l\" (UniqueName: \"kubernetes.io/projected/f41a94cc-0686-41ca-9666-2223a9736bd7-kube-api-access-6nw2l\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.107792 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-tls\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.108103 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-wtmp\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.108103 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107857 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f41a94cc-0686-41ca-9666-2223a9736bd7-sys\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.108103 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.107978 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-wtmp\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.108249 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.108124 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-textfile\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.108301 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.108248 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-accelerators-collector-config\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.108558 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.108512 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f41a94cc-0686-41ca-9666-2223a9736bd7-metrics-client-ca\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.110590 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.110566 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-tls\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.110702 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.110615 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f41a94cc-0686-41ca-9666-2223a9736bd7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.118958 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.118935 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nw2l\" (UniqueName: \"kubernetes.io/projected/f41a94cc-0686-41ca-9666-2223a9736bd7-kube-api-access-6nw2l\") pod \"node-exporter-qkq9g\" (UID: \"f41a94cc-0686-41ca-9666-2223a9736bd7\") " pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:07.235989 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:07.235957 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qkq9g" Jul 07 13:29:09.864641 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:29:09.864608 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41a94cc_0686_41ca_9666_2223a9736bd7.slice/crio-80317044b43f252f68e1d30b11c3b05f9cdbd4f79889950755ec83c6c49c6973 WatchSource:0}: Error finding container 80317044b43f252f68e1d30b11c3b05f9cdbd4f79889950755ec83c6c49c6973: Status 404 returned error can't find the container with id 80317044b43f252f68e1d30b11c3b05f9cdbd4f79889950755ec83c6c49c6973 Jul 07 13:29:10.068696 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:10.068664 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkq9g" event={"ID":"f41a94cc-0686-41ca-9666-2223a9736bd7","Type":"ContainerStarted","Data":"80317044b43f252f68e1d30b11c3b05f9cdbd4f79889950755ec83c6c49c6973"} Jul 07 13:29:11.074440 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:11.074398 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-zzfs7" event={"ID":"ac8db3f2-4183-4674-95b3-b7ea6ab06b58","Type":"ContainerStarted","Data":"ad2524835f264db8c5769c3ca50a0e9bb16fc22c2323e79112e536f7d83196bc"} Jul 07 13:29:11.074925 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:11.074621 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-zzfs7" Jul 07 13:29:11.091313 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:11.091288 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-zzfs7" Jul 07 13:29:11.096302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:11.096248 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-zzfs7" podStartSLOduration=1.367724092 podStartE2EDuration="20.096232352s" totalImagesPullingTime="18.72850826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:29:11.094028324 +0000 UTC m=+85.095269866" watchObservedRunningTime="2026-07-07 13:29:11.096232352 +0000 UTC m=+85.097473880" Jul 07 13:29:11.476956 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:11.476918 2576 patch_prober.go:28] interesting pod/image-registry-84dcfb865b-s2qps container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 13:29:11.477091 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:11.476986 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" podUID="b2bd6ca3-d88a-456f-913c-0ba058905a66" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 13:29:12.043753 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:12.043717 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xbmhz" Jul 07 13:29:12.080911 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:12.080872 2576 generic.go:356] "Generic (PLEG): container finished" podID="f41a94cc-0686-41ca-9666-2223a9736bd7" containerID="c5303619aa4bb4b487be26f92cf8f5bb76d99bc206390cb42deb05d4add22db8" exitCode=0 Jul 07 13:29:12.081551 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:12.080948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkq9g" event={"ID":"f41a94cc-0686-41ca-9666-2223a9736bd7","Type":"ContainerDied","Data":"c5303619aa4bb4b487be26f92cf8f5bb76d99bc206390cb42deb05d4add22db8"} Jul 07 13:29:12.988226 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:12.988193 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84dcfb865b-s2qps" Jul 07 13:29:13.088594 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:13.088549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkq9g" event={"ID":"f41a94cc-0686-41ca-9666-2223a9736bd7","Type":"ContainerStarted","Data":"70a3b9828f253e2fdf267a9be39fbb3f56e25df2480f01084d895c5f552b7f1e"} Jul 07 13:29:13.089072 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:13.088617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkq9g" event={"ID":"f41a94cc-0686-41ca-9666-2223a9736bd7","Type":"ContainerStarted","Data":"02725412b0f71e9649ffc96ec7dedb2a5e6e329be2de7dfd6dba4b9f626bcafc"} Jul 07 13:29:13.125186 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:13.125115 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qkq9g" podStartSLOduration=5.47178211 podStartE2EDuration="7.125094014s" totalImagesPullingTime="1.544233621s" totalInitContainerRuntime="109.078283ms" isStatefulPod=true podCreationTimestamp="2026-07-07 13:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:29:13.124256989 +0000 UTC m=+87.125498516" watchObservedRunningTime="2026-07-07 13:29:13.125094014 +0000 UTC m=+87.126335542" Jul 07 13:29:31.032331 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:29:31.032303 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tq7cd" Jul 07 13:32:46.408109 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:32:46.408082 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 13:33:25.260223 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.260190 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dfs72"] Jul 07 13:33:25.262159 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.262143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dfs72" Jul 07 13:33:25.265816 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.265753 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 13:33:25.266928 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.266910 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pfr5x\"" Jul 07 13:33:25.267055 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.266930 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 13:33:25.278003 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.277982 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dfs72"] Jul 07 13:33:25.338641 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.338614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v27pf\" (UniqueName: \"kubernetes.io/projected/9000623d-596a-4bdd-ace0-e4ea10c9ed61-kube-api-access-v27pf\") pod \"cert-manager-55b4d954d5-dfs72\" (UID: \"9000623d-596a-4bdd-ace0-e4ea10c9ed61\") " pod="cert-manager/cert-manager-55b4d954d5-dfs72" Jul 07 13:33:25.338739 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.338654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9000623d-596a-4bdd-ace0-e4ea10c9ed61-bound-sa-token\") pod \"cert-manager-55b4d954d5-dfs72\" (UID: \"9000623d-596a-4bdd-ace0-e4ea10c9ed61\") " pod="cert-manager/cert-manager-55b4d954d5-dfs72" Jul 07 13:33:25.439993 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.439957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v27pf\" (UniqueName: \"kubernetes.io/projected/9000623d-596a-4bdd-ace0-e4ea10c9ed61-kube-api-access-v27pf\") pod \"cert-manager-55b4d954d5-dfs72\" (UID: \"9000623d-596a-4bdd-ace0-e4ea10c9ed61\") " pod="cert-manager/cert-manager-55b4d954d5-dfs72" Jul 07 13:33:25.440151 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.440003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9000623d-596a-4bdd-ace0-e4ea10c9ed61-bound-sa-token\") pod \"cert-manager-55b4d954d5-dfs72\" (UID: \"9000623d-596a-4bdd-ace0-e4ea10c9ed61\") " pod="cert-manager/cert-manager-55b4d954d5-dfs72" Jul 07 13:33:25.451488 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.451457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9000623d-596a-4bdd-ace0-e4ea10c9ed61-bound-sa-token\") pod \"cert-manager-55b4d954d5-dfs72\" (UID: \"9000623d-596a-4bdd-ace0-e4ea10c9ed61\") " pod="cert-manager/cert-manager-55b4d954d5-dfs72" Jul 07 13:33:25.451597 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.451532 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v27pf\" (UniqueName: \"kubernetes.io/projected/9000623d-596a-4bdd-ace0-e4ea10c9ed61-kube-api-access-v27pf\") pod \"cert-manager-55b4d954d5-dfs72\" (UID: \"9000623d-596a-4bdd-ace0-e4ea10c9ed61\") " pod="cert-manager/cert-manager-55b4d954d5-dfs72" Jul 07 13:33:25.570257 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.570190 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dfs72" Jul 07 13:33:25.695896 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.695869 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dfs72"] Jul 07 13:33:25.698318 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:33:25.698283 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9000623d_596a_4bdd_ace0_e4ea10c9ed61.slice/crio-23f038074a7145b54978a57019e7ce1a889b898c9652dbcd75456a008d786950 WatchSource:0}: Error finding container 23f038074a7145b54978a57019e7ce1a889b898c9652dbcd75456a008d786950: Status 404 returned error can't find the container with id 23f038074a7145b54978a57019e7ce1a889b898c9652dbcd75456a008d786950 Jul 07 13:33:25.700080 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.700065 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 13:33:25.894696 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:25.894629 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dfs72" event={"ID":"9000623d-596a-4bdd-ace0-e4ea10c9ed61","Type":"ContainerStarted","Data":"23f038074a7145b54978a57019e7ce1a889b898c9652dbcd75456a008d786950"} Jul 07 13:33:27.760867 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.760835 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9"] Jul 07 13:33:27.763347 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.763322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" Jul 07 13:33:27.766505 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.766476 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 13:33:27.766505 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.766498 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 13:33:27.766696 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.766668 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-98qcm\"" Jul 07 13:33:27.778114 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.778089 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9"] Jul 07 13:33:27.859626 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.859593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57qg8\" (UniqueName: \"kubernetes.io/projected/dcc67eb5-eeea-4f74-b49a-7ced64b1be44-kube-api-access-57qg8\") pod \"openshift-lws-operator-fd8ccff4c-9h9d9\" (UID: \"dcc67eb5-eeea-4f74-b49a-7ced64b1be44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" Jul 07 13:33:27.859790 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.859648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcc67eb5-eeea-4f74-b49a-7ced64b1be44-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9h9d9\" (UID: \"dcc67eb5-eeea-4f74-b49a-7ced64b1be44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" Jul 07 13:33:27.960321 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.960287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57qg8\" (UniqueName: \"kubernetes.io/projected/dcc67eb5-eeea-4f74-b49a-7ced64b1be44-kube-api-access-57qg8\") pod \"openshift-lws-operator-fd8ccff4c-9h9d9\" (UID: \"dcc67eb5-eeea-4f74-b49a-7ced64b1be44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" Jul 07 13:33:27.960480 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.960344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcc67eb5-eeea-4f74-b49a-7ced64b1be44-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9h9d9\" (UID: \"dcc67eb5-eeea-4f74-b49a-7ced64b1be44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" Jul 07 13:33:27.960753 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.960732 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcc67eb5-eeea-4f74-b49a-7ced64b1be44-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9h9d9\" (UID: \"dcc67eb5-eeea-4f74-b49a-7ced64b1be44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" Jul 07 13:33:27.971188 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:27.971151 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57qg8\" (UniqueName: \"kubernetes.io/projected/dcc67eb5-eeea-4f74-b49a-7ced64b1be44-kube-api-access-57qg8\") pod \"openshift-lws-operator-fd8ccff4c-9h9d9\" (UID: \"dcc67eb5-eeea-4f74-b49a-7ced64b1be44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" Jul 07 13:33:28.074241 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:28.074145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" Jul 07 13:33:28.625049 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:28.625023 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9"] Jul 07 13:33:28.627464 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:33:28.627440 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc67eb5_eeea_4f74_b49a_7ced64b1be44.slice/crio-09e70443c38c632f8f79d0c344adcedecf6bdf7746bf64a13773d21df638b3fc WatchSource:0}: Error finding container 09e70443c38c632f8f79d0c344adcedecf6bdf7746bf64a13773d21df638b3fc: Status 404 returned error can't find the container with id 09e70443c38c632f8f79d0c344adcedecf6bdf7746bf64a13773d21df638b3fc Jul 07 13:33:28.907971 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:28.907886 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dfs72" event={"ID":"9000623d-596a-4bdd-ace0-e4ea10c9ed61","Type":"ContainerStarted","Data":"2656e91c3e73887962359d6f66fcc875c248194160b4e14022140a5d9b13a488"} Jul 07 13:33:28.909182 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:28.909145 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" event={"ID":"dcc67eb5-eeea-4f74-b49a-7ced64b1be44","Type":"ContainerStarted","Data":"09e70443c38c632f8f79d0c344adcedecf6bdf7746bf64a13773d21df638b3fc"} Jul 07 13:33:28.927391 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:28.927350 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dfs72" podStartSLOduration=1.09345995 podStartE2EDuration="3.927335409s" totalImagesPullingTime="2.833875459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:33:28.926411507 +0000 UTC m=+342.927653044" watchObservedRunningTime="2026-07-07 13:33:28.927335409 +0000 UTC m=+342.928576936" Jul 07 13:33:31.923093 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:31.923057 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" event={"ID":"dcc67eb5-eeea-4f74-b49a-7ced64b1be44","Type":"ContainerStarted","Data":"427509911c3e61d2644cc773ffb9a43d75f0cd4e59edc8b04ab2cacf21a7ce7f"} Jul 07 13:33:31.943974 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:31.943924 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9h9d9" podStartSLOduration=2.398356224 podStartE2EDuration="4.943908613s" totalImagesPullingTime="2.545552389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:33:31.9427358 +0000 UTC m=+345.943977329" watchObservedRunningTime="2026-07-07 13:33:31.943908613 +0000 UTC m=+345.945150140" Jul 07 13:33:47.726869 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.726792 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb"] Jul 07 13:33:47.729270 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.729253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.732736 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.732706 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 13:33:47.732867 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.732745 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 13:33:47.733699 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.733682 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 13:33:47.734157 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.734137 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 13:33:47.734285 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.734153 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nb9q7\"" Jul 07 13:33:47.782743 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.782720 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb"] Jul 07 13:33:47.804451 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.804430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkrn7\" (UniqueName: \"kubernetes.io/projected/3f990869-3802-4dbb-b41b-15e6832bf9a1-kube-api-access-vkrn7\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.804560 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.804467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f990869-3802-4dbb-b41b-15e6832bf9a1-tmp\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.804560 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.804492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f990869-3802-4dbb-b41b-15e6832bf9a1-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.804560 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.804514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f990869-3802-4dbb-b41b-15e6832bf9a1-manifests\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.804698 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.804574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f990869-3802-4dbb-b41b-15e6832bf9a1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.905205 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.905158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f990869-3802-4dbb-b41b-15e6832bf9a1-manifests\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.905205 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.905207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f990869-3802-4dbb-b41b-15e6832bf9a1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.905431 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.905247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkrn7\" (UniqueName: \"kubernetes.io/projected/3f990869-3802-4dbb-b41b-15e6832bf9a1-kube-api-access-vkrn7\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.905431 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.905277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f990869-3802-4dbb-b41b-15e6832bf9a1-tmp\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.905431 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.905302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f990869-3802-4dbb-b41b-15e6832bf9a1-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.905856 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.905549 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f990869-3802-4dbb-b41b-15e6832bf9a1-manifests\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.907498 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.907469 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f990869-3802-4dbb-b41b-15e6832bf9a1-tmp\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.907702 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.907683 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f990869-3802-4dbb-b41b-15e6832bf9a1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.907764 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.907727 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f990869-3802-4dbb-b41b-15e6832bf9a1-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:47.915659 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:47.915636 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkrn7\" (UniqueName: \"kubernetes.io/projected/3f990869-3802-4dbb-b41b-15e6832bf9a1-kube-api-access-vkrn7\") pod \"opendatahub-operator-controller-manager-76f9489749-nlmcb\" (UID: \"3f990869-3802-4dbb-b41b-15e6832bf9a1\") " pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:48.039595 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:48.039519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:33:48.174197 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:48.174160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb"] Jul 07 13:33:48.176464 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:33:48.176436 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f990869_3802_4dbb_b41b_15e6832bf9a1.slice/crio-3b170f1d349cf2961132ecaef67dd4031fe6990cc90003b8364fc0803da9d624 WatchSource:0}: Error finding container 3b170f1d349cf2961132ecaef67dd4031fe6990cc90003b8364fc0803da9d624: Status 404 returned error can't find the container with id 3b170f1d349cf2961132ecaef67dd4031fe6990cc90003b8364fc0803da9d624 Jul 07 13:33:48.981805 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:48.981762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" event={"ID":"3f990869-3802-4dbb-b41b-15e6832bf9a1","Type":"ContainerStarted","Data":"3b170f1d349cf2961132ecaef67dd4031fe6990cc90003b8364fc0803da9d624"} Jul 07 13:33:51.995976 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:51.995940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" event={"ID":"3f990869-3802-4dbb-b41b-15e6832bf9a1","Type":"ContainerStarted","Data":"014b1905d960a5287b5904df546e8a26d17e3ca86de62868ef39d6f7158602ba"} Jul 07 13:33:53.000828 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:53.000795 2576 generic.go:356] "Generic (PLEG): container finished" podID="3f990869-3802-4dbb-b41b-15e6832bf9a1" containerID="014b1905d960a5287b5904df546e8a26d17e3ca86de62868ef39d6f7158602ba" exitCode=0 Jul 07 13:33:53.001276 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:53.000846 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" event={"ID":"3f990869-3802-4dbb-b41b-15e6832bf9a1","Type":"ContainerDied","Data":"014b1905d960a5287b5904df546e8a26d17e3ca86de62868ef39d6f7158602ba"} Jul 07 13:33:54.006619 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:54.006582 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" event={"ID":"3f990869-3802-4dbb-b41b-15e6832bf9a1","Type":"ContainerStarted","Data":"ebab49983c95b902a51e8a68913695447f21ddb3efe599607d62230be489e56d"} Jul 07 13:33:54.007083 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:33:54.006694 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:34:05.013398 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:05.013369 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" Jul 07 13:34:05.041741 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:05.041696 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76f9489749-nlmcb" podStartSLOduration=13.603922444 podStartE2EDuration="18.041679808s" totalImagesPullingTime="3.385268649s" totalInitContainerRuntime="1.052488715s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:33:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:33:54.033837865 +0000 UTC m=+368.035079392" watchObservedRunningTime="2026-07-07 13:34:05.041679808 +0000 UTC m=+379.042921335" Jul 07 13:34:08.283019 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.282989 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9s5wf"] Jul 07 13:34:08.284981 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.284966 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" Jul 07 13:34:08.288379 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.288362 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 07 13:34:08.297957 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.297937 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9s5wf"] Jul 07 13:34:08.458457 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.458428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvsvg\" (UniqueName: \"kubernetes.io/projected/865a955f-a66c-426e-a54b-05dd7cb5c85e-kube-api-access-mvsvg\") pod \"model-serving-api-54d89bd5fc-9s5wf\" (UID: \"865a955f-a66c-426e-a54b-05dd7cb5c85e\") " pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" Jul 07 13:34:08.458608 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.458472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865a955f-a66c-426e-a54b-05dd7cb5c85e-tls-certs\") pod \"model-serving-api-54d89bd5fc-9s5wf\" (UID: \"865a955f-a66c-426e-a54b-05dd7cb5c85e\") " pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" Jul 07 13:34:08.559764 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.559693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvsvg\" (UniqueName: \"kubernetes.io/projected/865a955f-a66c-426e-a54b-05dd7cb5c85e-kube-api-access-mvsvg\") pod \"model-serving-api-54d89bd5fc-9s5wf\" (UID: \"865a955f-a66c-426e-a54b-05dd7cb5c85e\") " pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" Jul 07 13:34:08.559764 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.559735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865a955f-a66c-426e-a54b-05dd7cb5c85e-tls-certs\") pod \"model-serving-api-54d89bd5fc-9s5wf\" (UID: \"865a955f-a66c-426e-a54b-05dd7cb5c85e\") " pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" Jul 07 13:34:08.562211 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.562189 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865a955f-a66c-426e-a54b-05dd7cb5c85e-tls-certs\") pod \"model-serving-api-54d89bd5fc-9s5wf\" (UID: \"865a955f-a66c-426e-a54b-05dd7cb5c85e\") " pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" Jul 07 13:34:08.573330 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.573308 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvsvg\" (UniqueName: \"kubernetes.io/projected/865a955f-a66c-426e-a54b-05dd7cb5c85e-kube-api-access-mvsvg\") pod \"model-serving-api-54d89bd5fc-9s5wf\" (UID: \"865a955f-a66c-426e-a54b-05dd7cb5c85e\") " pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" Jul 07 13:34:08.593752 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.593732 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" Jul 07 13:34:08.724778 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:08.724689 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9s5wf"] Jul 07 13:34:08.727427 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:34:08.727399 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865a955f_a66c_426e_a54b_05dd7cb5c85e.slice/crio-b15a964d31c3ef96701637311c37e796ca71a5f47c2bb58b7969b4bdc10c0ada WatchSource:0}: Error finding container b15a964d31c3ef96701637311c37e796ca71a5f47c2bb58b7969b4bdc10c0ada: Status 404 returned error can't find the container with id b15a964d31c3ef96701637311c37e796ca71a5f47c2bb58b7969b4bdc10c0ada Jul 07 13:34:09.064776 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:09.064742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" event={"ID":"865a955f-a66c-426e-a54b-05dd7cb5c85e","Type":"ContainerStarted","Data":"b15a964d31c3ef96701637311c37e796ca71a5f47c2bb58b7969b4bdc10c0ada"} Jul 07 13:34:11.074716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:11.074680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" event={"ID":"865a955f-a66c-426e-a54b-05dd7cb5c85e","Type":"ContainerStarted","Data":"80908d477d721458048d2678908b10f5596c696bfdf5866b50bf08b2ab52056d"} Jul 07 13:34:11.075088 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:11.074820 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" Jul 07 13:34:11.092524 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:11.092480 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" podStartSLOduration=1.124259795 podStartE2EDuration="3.092467808s" totalImagesPullingTime="1.968208013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:34:11.092045157 +0000 UTC m=+385.093286683" watchObservedRunningTime="2026-07-07 13:34:11.092467808 +0000 UTC m=+385.093709335" Jul 07 13:34:12.886655 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.886621 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w"] Jul 07 13:34:12.889562 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.889546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:12.895272 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.895248 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m875t\"" Jul 07 13:34:12.895411 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.895321 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 13:34:12.895411 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.895342 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 13:34:12.895411 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.895366 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 13:34:12.895635 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.895617 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 13:34:12.895826 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.895810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlgm\" (UniqueName: \"kubernetes.io/projected/63401851-59e7-4ee7-9df8-73148808810d-kube-api-access-2tlgm\") pod \"kube-auth-proxy-87699d5c5-6zj4w\" (UID: \"63401851-59e7-4ee7-9df8-73148808810d\") " pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:12.895900 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.895844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63401851-59e7-4ee7-9df8-73148808810d-tmp\") pod \"kube-auth-proxy-87699d5c5-6zj4w\" (UID: \"63401851-59e7-4ee7-9df8-73148808810d\") " pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:12.895900 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.895883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63401851-59e7-4ee7-9df8-73148808810d-tls-certs\") pod \"kube-auth-proxy-87699d5c5-6zj4w\" (UID: \"63401851-59e7-4ee7-9df8-73148808810d\") " pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:12.903041 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.903017 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w"] Jul 07 13:34:12.996580 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.996553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63401851-59e7-4ee7-9df8-73148808810d-tmp\") pod \"kube-auth-proxy-87699d5c5-6zj4w\" (UID: \"63401851-59e7-4ee7-9df8-73148808810d\") " pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:12.996757 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.996598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63401851-59e7-4ee7-9df8-73148808810d-tls-certs\") pod \"kube-auth-proxy-87699d5c5-6zj4w\" (UID: \"63401851-59e7-4ee7-9df8-73148808810d\") " pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:12.996757 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.996651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlgm\" (UniqueName: \"kubernetes.io/projected/63401851-59e7-4ee7-9df8-73148808810d-kube-api-access-2tlgm\") pod \"kube-auth-proxy-87699d5c5-6zj4w\" (UID: \"63401851-59e7-4ee7-9df8-73148808810d\") " pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:12.998790 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.998762 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63401851-59e7-4ee7-9df8-73148808810d-tmp\") pod \"kube-auth-proxy-87699d5c5-6zj4w\" (UID: \"63401851-59e7-4ee7-9df8-73148808810d\") " pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:12.998918 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:12.998899 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63401851-59e7-4ee7-9df8-73148808810d-tls-certs\") pod \"kube-auth-proxy-87699d5c5-6zj4w\" (UID: \"63401851-59e7-4ee7-9df8-73148808810d\") " pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:13.009645 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:13.009626 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlgm\" (UniqueName: \"kubernetes.io/projected/63401851-59e7-4ee7-9df8-73148808810d-kube-api-access-2tlgm\") pod \"kube-auth-proxy-87699d5c5-6zj4w\" (UID: \"63401851-59e7-4ee7-9df8-73148808810d\") " pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:13.198971 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:13.198901 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" Jul 07 13:34:13.320701 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:13.320672 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w"] Jul 07 13:34:13.323242 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:34:13.323217 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63401851_59e7_4ee7_9df8_73148808810d.slice/crio-9897efa2fc05eec9a8377d1bbbeaca5cf5941dbfd52f5dd3cbbdeb4fd038d672 WatchSource:0}: Error finding container 9897efa2fc05eec9a8377d1bbbeaca5cf5941dbfd52f5dd3cbbdeb4fd038d672: Status 404 returned error can't find the container with id 9897efa2fc05eec9a8377d1bbbeaca5cf5941dbfd52f5dd3cbbdeb4fd038d672 Jul 07 13:34:14.087446 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:14.087412 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" event={"ID":"63401851-59e7-4ee7-9df8-73148808810d","Type":"ContainerStarted","Data":"9897efa2fc05eec9a8377d1bbbeaca5cf5941dbfd52f5dd3cbbdeb4fd038d672"} Jul 07 13:34:14.914493 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:14.914458 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5"] Jul 07 13:34:14.916744 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:14.916709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:14.919703 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:14.919675 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 13:34:14.920850 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:14.920828 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hrx5l\"" Jul 07 13:34:14.921001 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:14.920984 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 13:34:14.921001 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:14.920997 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 13:34:14.921964 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:14.921947 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 13:34:14.942737 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:14.942708 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5"] Jul 07 13:34:15.011371 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.011160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-istio-token\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.011371 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.011267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77cv5\" (UniqueName: \"kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-kube-api-access-77cv5\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.011371 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.011331 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.011371 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.011355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6803b783-ed7c-4369-b709-e058f1a3754d-local-certs\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.011708 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.011402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.011708 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.011466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-cacerts\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.011708 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.011517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.112795 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.112761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.113302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.112811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-istio-token\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.113302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.112858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77cv5\" (UniqueName: \"kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-kube-api-access-77cv5\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.113302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.112886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.113302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.112909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6803b783-ed7c-4369-b709-e058f1a3754d-local-certs\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.113302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.112935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.113302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.112959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-cacerts\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.113877 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.113849 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.115602 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.115555 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.115703 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.115675 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-cacerts\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.115703 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.115683 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6803b783-ed7c-4369-b709-e058f1a3754d-local-certs\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.116032 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.116014 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.122574 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.122546 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-istio-token\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.122970 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.122949 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77cv5\" (UniqueName: \"kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-kube-api-access-77cv5\") pod \"istiod-openshift-gateway-7574459656-g5wf5\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.226907 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.226879 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:15.395491 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:15.395406 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5"] Jul 07 13:34:15.397541 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:34:15.397513 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6803b783_ed7c_4369_b709_e058f1a3754d.slice/crio-6d5dcb22fc0973b95f39164ead103f66047b6f05e7866bddd8b32b8c453c1dda WatchSource:0}: Error finding container 6d5dcb22fc0973b95f39164ead103f66047b6f05e7866bddd8b32b8c453c1dda: Status 404 returned error can't find the container with id 6d5dcb22fc0973b95f39164ead103f66047b6f05e7866bddd8b32b8c453c1dda Jul 07 13:34:16.097435 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:16.097390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" event={"ID":"6803b783-ed7c-4369-b709-e058f1a3754d","Type":"ContainerStarted","Data":"6d5dcb22fc0973b95f39164ead103f66047b6f05e7866bddd8b32b8c453c1dda"} Jul 07 13:34:17.102364 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:17.102324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" event={"ID":"63401851-59e7-4ee7-9df8-73148808810d","Type":"ContainerStarted","Data":"30db65427abaf9284afe00cbd71ef2bb903127009236772091aced62dbe1edad"} Jul 07 13:34:17.163494 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:17.163444 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-87699d5c5-6zj4w" podStartSLOduration=2.152632532 podStartE2EDuration="5.163430784s" totalImagesPullingTime="3.010798252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:34:17.159622011 +0000 UTC m=+391.160863537" watchObservedRunningTime="2026-07-07 13:34:17.163430784 +0000 UTC m=+391.164672311" Jul 07 13:34:19.687048 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.687013 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg"] Jul 07 13:34:19.689478 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.689463 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:19.694533 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.694511 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-smkhc\"" Jul 07 13:34:19.701266 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.701247 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 07 13:34:19.707720 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.707701 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg"] Jul 07 13:34:19.710556 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.710536 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x"] Jul 07 13:34:19.717010 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.716995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" Jul 07 13:34:19.742589 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.742569 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x"] Jul 07 13:34:19.744967 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.744946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k5sq\" (UniqueName: \"kubernetes.io/projected/e07431ca-19e6-405b-8303-174e7a32d40a-kube-api-access-4k5sq\") pod \"kserve-controller-manager-d6f4b65b8-tx2mg\" (UID: \"e07431ca-19e6-405b-8303-174e7a32d40a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:19.745071 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.745004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42e7da0e-84bc-4757-bccc-26d24e0bd443-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qmp7x\" (UID: \"42e7da0e-84bc-4757-bccc-26d24e0bd443\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" Jul 07 13:34:19.745071 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.745060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e07431ca-19e6-405b-8303-174e7a32d40a-cert\") pod \"kserve-controller-manager-d6f4b65b8-tx2mg\" (UID: \"e07431ca-19e6-405b-8303-174e7a32d40a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:19.745160 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.745085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmwq\" (UniqueName: \"kubernetes.io/projected/42e7da0e-84bc-4757-bccc-26d24e0bd443-kube-api-access-hmmwq\") pod \"llmisvc-controller-manager-777c6bd7d5-qmp7x\" (UID: \"42e7da0e-84bc-4757-bccc-26d24e0bd443\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" Jul 07 13:34:19.757889 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.757868 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cw77l\"" Jul 07 13:34:19.758619 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.758606 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 07 13:34:19.845824 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.845798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k5sq\" (UniqueName: \"kubernetes.io/projected/e07431ca-19e6-405b-8303-174e7a32d40a-kube-api-access-4k5sq\") pod \"kserve-controller-manager-d6f4b65b8-tx2mg\" (UID: \"e07431ca-19e6-405b-8303-174e7a32d40a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:19.845955 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.845848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42e7da0e-84bc-4757-bccc-26d24e0bd443-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qmp7x\" (UID: \"42e7da0e-84bc-4757-bccc-26d24e0bd443\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" Jul 07 13:34:19.845955 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.845880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e07431ca-19e6-405b-8303-174e7a32d40a-cert\") pod \"kserve-controller-manager-d6f4b65b8-tx2mg\" (UID: \"e07431ca-19e6-405b-8303-174e7a32d40a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:19.846023 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:34:19.845962 2576 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 07 13:34:19.846023 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:34:19.846015 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e07431ca-19e6-405b-8303-174e7a32d40a-cert podName:e07431ca-19e6-405b-8303-174e7a32d40a nodeName:}" failed. No retries permitted until 2026-07-07 13:34:20.345996792 +0000 UTC m=+394.347238297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e07431ca-19e6-405b-8303-174e7a32d40a-cert") pod "kserve-controller-manager-d6f4b65b8-tx2mg" (UID: "e07431ca-19e6-405b-8303-174e7a32d40a") : secret "kserve-webhook-server-cert" not found Jul 07 13:34:19.846102 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.846046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmwq\" (UniqueName: \"kubernetes.io/projected/42e7da0e-84bc-4757-bccc-26d24e0bd443-kube-api-access-hmmwq\") pod \"llmisvc-controller-manager-777c6bd7d5-qmp7x\" (UID: \"42e7da0e-84bc-4757-bccc-26d24e0bd443\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" Jul 07 13:34:19.848209 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.848192 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42e7da0e-84bc-4757-bccc-26d24e0bd443-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qmp7x\" (UID: \"42e7da0e-84bc-4757-bccc-26d24e0bd443\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" Jul 07 13:34:19.866127 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.866105 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k5sq\" (UniqueName: \"kubernetes.io/projected/e07431ca-19e6-405b-8303-174e7a32d40a-kube-api-access-4k5sq\") pod \"kserve-controller-manager-d6f4b65b8-tx2mg\" (UID: \"e07431ca-19e6-405b-8303-174e7a32d40a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:19.892196 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:19.892154 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmwq\" (UniqueName: \"kubernetes.io/projected/42e7da0e-84bc-4757-bccc-26d24e0bd443-kube-api-access-hmmwq\") pod \"llmisvc-controller-manager-777c6bd7d5-qmp7x\" (UID: \"42e7da0e-84bc-4757-bccc-26d24e0bd443\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" Jul 07 13:34:20.026745 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:20.026711 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" Jul 07 13:34:20.207694 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:34:20.207667 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod42e7da0e_84bc_4757_bccc_26d24e0bd443.slice/crio-d222c946fec5781b00c767afff2cd42ae2617e5ada96fcbf3ca0dd13ba076b3f WatchSource:0}: Error finding container d222c946fec5781b00c767afff2cd42ae2617e5ada96fcbf3ca0dd13ba076b3f: Status 404 returned error can't find the container with id d222c946fec5781b00c767afff2cd42ae2617e5ada96fcbf3ca0dd13ba076b3f Jul 07 13:34:20.210717 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:20.210692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x"] Jul 07 13:34:20.350204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:20.350116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e07431ca-19e6-405b-8303-174e7a32d40a-cert\") pod \"kserve-controller-manager-d6f4b65b8-tx2mg\" (UID: \"e07431ca-19e6-405b-8303-174e7a32d40a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:20.352423 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:20.352396 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e07431ca-19e6-405b-8303-174e7a32d40a-cert\") pod \"kserve-controller-manager-d6f4b65b8-tx2mg\" (UID: \"e07431ca-19e6-405b-8303-174e7a32d40a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:20.598755 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:20.598724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:20.744407 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:20.744373 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg"] Jul 07 13:34:20.748033 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:34:20.748007 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07431ca_19e6_405b_8303_174e7a32d40a.slice/crio-2ae1e2ceba2e972c14c59cdaaf58307ff86135d0ce8e8fe0c64f98837307cd66 WatchSource:0}: Error finding container 2ae1e2ceba2e972c14c59cdaaf58307ff86135d0ce8e8fe0c64f98837307cd66: Status 404 returned error can't find the container with id 2ae1e2ceba2e972c14c59cdaaf58307ff86135d0ce8e8fe0c64f98837307cd66 Jul 07 13:34:21.137064 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:21.137024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" event={"ID":"e07431ca-19e6-405b-8303-174e7a32d40a","Type":"ContainerStarted","Data":"2ae1e2ceba2e972c14c59cdaaf58307ff86135d0ce8e8fe0c64f98837307cd66"} Jul 07 13:34:21.138329 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:21.138295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" event={"ID":"42e7da0e-84bc-4757-bccc-26d24e0bd443","Type":"ContainerStarted","Data":"d222c946fec5781b00c767afff2cd42ae2617e5ada96fcbf3ca0dd13ba076b3f"} Jul 07 13:34:21.987576 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:21.987269 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp"] Jul 07 13:34:21.993400 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:21.993377 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:21.999762 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:21.999735 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gqnmw\"" Jul 07 13:34:22.045985 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.045912 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp"] Jul 07 13:34:22.064053 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.063968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.064053 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.064013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.064053 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.064044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkhb\" (UniqueName: \"kubernetes.io/projected/031f9d9e-0d19-4198-a277-3c7a8e4d5469-kube-api-access-xvkhb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.064366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.064128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.064366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.064192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.064366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.064226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.064366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.064281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.064366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.064311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.064366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.064347 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.084335 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.084308 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-9s5wf" Jul 07 13:34:22.165285 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165237 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.165467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.165467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkhb\" (UniqueName: \"kubernetes.io/projected/031f9d9e-0d19-4198-a277-3c7a8e4d5469-kube-api-access-xvkhb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.165467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.165467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.165467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.165739 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.165739 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.165739 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165566 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.165739 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.165739 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.165735 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.166751 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.166233 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.166751 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.166677 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.166751 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.166692 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.168601 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.168564 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.168705 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.168624 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.183618 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.183597 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031f9d9e-0d19-4198-a277-3c7a8e4d5469-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.187521 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.187489 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkhb\" (UniqueName: \"kubernetes.io/projected/031f9d9e-0d19-4198-a277-3c7a8e4d5469-kube-api-access-xvkhb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4s7csp\" (UID: \"031f9d9e-0d19-4198-a277-3c7a8e4d5469\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:22.308720 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:22.308636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:23.101252 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:23.101213 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp"] Jul 07 13:34:23.513862 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:34:23.513828 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031f9d9e_0d19_4198_a277_3c7a8e4d5469.slice/crio-25afe7dc8b2d02a43445df5df179dc308e7c6984a7f8fb7205adfd5f58fdf49f WatchSource:0}: Error finding container 25afe7dc8b2d02a43445df5df179dc308e7c6984a7f8fb7205adfd5f58fdf49f: Status 404 returned error can't find the container with id 25afe7dc8b2d02a43445df5df179dc308e7c6984a7f8fb7205adfd5f58fdf49f Jul 07 13:34:23.871444 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:23.871364 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5"] Jul 07 13:34:24.077013 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:24.076962 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:34:24.077209 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:24.077099 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:34:24.154308 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:24.154280 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" event={"ID":"6803b783-ed7c-4369-b709-e058f1a3754d","Type":"ContainerStarted","Data":"ff4bd5d157e9656b47d3ff5b4a9c716058e7b3514d512bdf157859b78821ba9e"} Jul 07 13:34:24.154678 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:24.154340 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" podUID="6803b783-ed7c-4369-b709-e058f1a3754d" containerName="discovery" containerID="cri-o://ff4bd5d157e9656b47d3ff5b4a9c716058e7b3514d512bdf157859b78821ba9e" gracePeriod=30 Jul 07 13:34:24.154678 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:24.154407 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:24.155823 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:24.155797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" event={"ID":"031f9d9e-0d19-4198-a277-3c7a8e4d5469","Type":"ContainerStarted","Data":"25afe7dc8b2d02a43445df5df179dc308e7c6984a7f8fb7205adfd5f58fdf49f"} Jul 07 13:34:24.157625 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:24.157611 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:24.223252 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:24.223201 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" podStartSLOduration=1.546031162 podStartE2EDuration="10.22318523s" totalImagesPullingTime="8.677154068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:34:24.222790101 +0000 UTC m=+398.224031627" watchObservedRunningTime="2026-07-07 13:34:24.22318523 +0000 UTC m=+398.224426754" Jul 07 13:34:25.163900 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:25.163865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" event={"ID":"e07431ca-19e6-405b-8303-174e7a32d40a","Type":"ContainerStarted","Data":"9538619fef9e335b2707cfb8d995a390e47906bf0a30d7027ccab2c9366b71ce"} Jul 07 13:34:25.165600 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:25.165565 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" event={"ID":"42e7da0e-84bc-4757-bccc-26d24e0bd443","Type":"ContainerStarted","Data":"1dfb15d4d94e3a46ecbbeec1ed2af8c47a5c0684692df8c74fda7b0b36824cec"} Jul 07 13:34:25.165737 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:25.165655 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" Jul 07 13:34:25.208632 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:25.208588 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" podStartSLOduration=2.333026366 podStartE2EDuration="6.208572171s" totalImagesPullingTime="3.875545805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:34:25.207835413 +0000 UTC m=+399.209076941" watchObservedRunningTime="2026-07-07 13:34:25.208572171 +0000 UTC m=+399.209813698" Jul 07 13:34:25.208809 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:25.208779 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" podStartSLOduration=2.874066692 podStartE2EDuration="6.208771098s" totalImagesPullingTime="3.334704406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:34:24.276547697 +0000 UTC m=+398.277789225" watchObservedRunningTime="2026-07-07 13:34:25.208771098 +0000 UTC m=+399.210012623" Jul 07 13:34:26.717740 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:26.717699 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:34:26.718024 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:26.717831 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:34:26.718024 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:26.717867 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:34:27.156986 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:27.156956 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:27.175208 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:27.175159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" event={"ID":"031f9d9e-0d19-4198-a277-3c7a8e4d5469","Type":"ContainerStarted","Data":"96485e618c286bffe41e1cefa0838d72a82e005c3b8d391fbcb7f27fe7547905"} Jul 07 13:34:27.206831 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:27.206782 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" podStartSLOduration=3.006169825 podStartE2EDuration="6.206767461s" totalImagesPullingTime="3.200597636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:34:27.204687014 +0000 UTC m=+401.205928553" watchObservedRunningTime="2026-07-07 13:34:27.206767461 +0000 UTC m=+401.208008988" Jul 07 13:34:27.309201 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:27.309156 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:27.313691 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:27.313669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:28.179710 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:28.179680 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:28.180615 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:28.180599 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4s7csp" Jul 07 13:34:54.287761 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.287731 2576 generic.go:356] "Generic (PLEG): container finished" podID="6803b783-ed7c-4369-b709-e058f1a3754d" containerID="ff4bd5d157e9656b47d3ff5b4a9c716058e7b3514d512bdf157859b78821ba9e" exitCode=137 Jul 07 13:34:54.288130 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.287788 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" event={"ID":"6803b783-ed7c-4369-b709-e058f1a3754d","Type":"ContainerDied","Data":"ff4bd5d157e9656b47d3ff5b4a9c716058e7b3514d512bdf157859b78821ba9e"} Jul 07 13:34:54.798364 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.798341 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:54.916403 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.916378 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-kubeconfig\") pod \"6803b783-ed7c-4369-b709-e058f1a3754d\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " Jul 07 13:34:54.916403 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.916408 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-cacerts\") pod \"6803b783-ed7c-4369-b709-e058f1a3754d\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " Jul 07 13:34:54.916616 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.916435 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-ca-configmap\") pod \"6803b783-ed7c-4369-b709-e058f1a3754d\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " Jul 07 13:34:54.916616 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.916492 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-istio-token\") pod \"6803b783-ed7c-4369-b709-e058f1a3754d\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " Jul 07 13:34:54.916616 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.916534 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6803b783-ed7c-4369-b709-e058f1a3754d-local-certs\") pod \"6803b783-ed7c-4369-b709-e058f1a3754d\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " Jul 07 13:34:54.916616 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.916571 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77cv5\" (UniqueName: \"kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-kube-api-access-77cv5\") pod \"6803b783-ed7c-4369-b709-e058f1a3754d\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " Jul 07 13:34:54.916616 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.916599 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-dns-cert\") pod \"6803b783-ed7c-4369-b709-e058f1a3754d\" (UID: \"6803b783-ed7c-4369-b709-e058f1a3754d\") " Jul 07 13:34:54.916914 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.916885 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "6803b783-ed7c-4369-b709-e058f1a3754d" (UID: "6803b783-ed7c-4369-b709-e058f1a3754d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:34:54.918790 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.918675 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-istio-token" (OuterVolumeSpecName: "istio-token") pod "6803b783-ed7c-4369-b709-e058f1a3754d" (UID: "6803b783-ed7c-4369-b709-e058f1a3754d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:34:54.918790 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.918731 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "6803b783-ed7c-4369-b709-e058f1a3754d" (UID: "6803b783-ed7c-4369-b709-e058f1a3754d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 13:34:54.919026 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.918844 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-cacerts" (OuterVolumeSpecName: "cacerts") pod "6803b783-ed7c-4369-b709-e058f1a3754d" (UID: "6803b783-ed7c-4369-b709-e058f1a3754d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 13:34:54.919026 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.918858 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6803b783-ed7c-4369-b709-e058f1a3754d-local-certs" (OuterVolumeSpecName: "local-certs") pod "6803b783-ed7c-4369-b709-e058f1a3754d" (UID: "6803b783-ed7c-4369-b709-e058f1a3754d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:34:54.919139 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.919039 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "6803b783-ed7c-4369-b709-e058f1a3754d" (UID: "6803b783-ed7c-4369-b709-e058f1a3754d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 13:34:54.919139 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:54.919041 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-kube-api-access-77cv5" (OuterVolumeSpecName: "kube-api-access-77cv5") pod "6803b783-ed7c-4369-b709-e058f1a3754d" (UID: "6803b783-ed7c-4369-b709-e058f1a3754d"). InnerVolumeSpecName "kube-api-access-77cv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:34:55.017472 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.017442 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:34:55.017472 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.017466 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6803b783-ed7c-4369-b709-e058f1a3754d-local-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:34:55.017472 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.017475 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77cv5\" (UniqueName: \"kubernetes.io/projected/6803b783-ed7c-4369-b709-e058f1a3754d-kube-api-access-77cv5\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:34:55.017672 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.017483 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-dns-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:34:55.017672 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.017492 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-istio-kubeconfig\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:34:55.017672 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.017500 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6803b783-ed7c-4369-b709-e058f1a3754d-cacerts\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:34:55.017672 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.017508 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6803b783-ed7c-4369-b709-e058f1a3754d-istio-csr-ca-configmap\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:34:55.292849 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.292817 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" event={"ID":"6803b783-ed7c-4369-b709-e058f1a3754d","Type":"ContainerDied","Data":"6d5dcb22fc0973b95f39164ead103f66047b6f05e7866bddd8b32b8c453c1dda"} Jul 07 13:34:55.292849 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.292841 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5" Jul 07 13:34:55.293354 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.292870 2576 scope.go:117] "RemoveContainer" containerID="ff4bd5d157e9656b47d3ff5b4a9c716058e7b3514d512bdf157859b78821ba9e" Jul 07 13:34:55.315288 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.315267 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5"] Jul 07 13:34:55.320575 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:55.320556 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g5wf5"] Jul 07 13:34:56.172015 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:56.171984 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qmp7x" Jul 07 13:34:56.175498 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:56.175479 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tx2mg" Jul 07 13:34:56.589216 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:34:56.589186 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6803b783-ed7c-4369-b709-e058f1a3754d" path="/var/lib/kubelet/pods/6803b783-ed7c-4369-b709-e058f1a3754d/volumes" Jul 07 13:35:08.398528 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.398448 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f5789"] Jul 07 13:35:08.398991 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.398779 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6803b783-ed7c-4369-b709-e058f1a3754d" containerName="discovery" Jul 07 13:35:08.398991 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.398792 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6803b783-ed7c-4369-b709-e058f1a3754d" containerName="discovery" Jul 07 13:35:08.398991 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.398847 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6803b783-ed7c-4369-b709-e058f1a3754d" containerName="discovery" Jul 07 13:35:08.400592 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.400571 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f5789" Jul 07 13:35:08.404573 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.404543 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 13:35:08.405822 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.405800 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-kk5zb\"" Jul 07 13:35:08.406243 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.405808 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 13:35:08.413752 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.413730 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f5789"] Jul 07 13:35:08.526542 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.526504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8gk\" (UniqueName: \"kubernetes.io/projected/e7bafb5c-cc08-4291-aff4-4f9989a352b6-kube-api-access-gt8gk\") pod \"kuadrant-operator-catalog-f5789\" (UID: \"e7bafb5c-cc08-4291-aff4-4f9989a352b6\") " pod="kuadrant-system/kuadrant-operator-catalog-f5789" Jul 07 13:35:08.626978 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.626945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8gk\" (UniqueName: \"kubernetes.io/projected/e7bafb5c-cc08-4291-aff4-4f9989a352b6-kube-api-access-gt8gk\") pod \"kuadrant-operator-catalog-f5789\" (UID: \"e7bafb5c-cc08-4291-aff4-4f9989a352b6\") " pod="kuadrant-system/kuadrant-operator-catalog-f5789" Jul 07 13:35:08.636496 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.636469 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8gk\" (UniqueName: \"kubernetes.io/projected/e7bafb5c-cc08-4291-aff4-4f9989a352b6-kube-api-access-gt8gk\") pod \"kuadrant-operator-catalog-f5789\" (UID: \"e7bafb5c-cc08-4291-aff4-4f9989a352b6\") " pod="kuadrant-system/kuadrant-operator-catalog-f5789" Jul 07 13:35:08.637548 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.637527 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f5789"] Jul 07 13:35:08.637778 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.637767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f5789" Jul 07 13:35:08.788379 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:35:08.788350 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bafb5c_cc08_4291_aff4_4f9989a352b6.slice/crio-53e606459f7c2e60ddb40cedd5e3644e3109a548c4831ae030600edad5692180 WatchSource:0}: Error finding container 53e606459f7c2e60ddb40cedd5e3644e3109a548c4831ae030600edad5692180: Status 404 returned error can't find the container with id 53e606459f7c2e60ddb40cedd5e3644e3109a548c4831ae030600edad5692180 Jul 07 13:35:08.807053 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.807027 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f5789"] Jul 07 13:35:08.897941 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.897910 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-w496l"] Jul 07 13:35:08.900965 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.900922 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-w496l" Jul 07 13:35:08.914490 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:08.914465 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-w496l"] Jul 07 13:35:09.029736 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:09.029707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4lkq\" (UniqueName: \"kubernetes.io/projected/f763fb5c-d2d5-4832-b013-109b98dc92fb-kube-api-access-s4lkq\") pod \"kuadrant-operator-catalog-w496l\" (UID: \"f763fb5c-d2d5-4832-b013-109b98dc92fb\") " pod="kuadrant-system/kuadrant-operator-catalog-w496l" Jul 07 13:35:09.130864 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:09.130826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4lkq\" (UniqueName: \"kubernetes.io/projected/f763fb5c-d2d5-4832-b013-109b98dc92fb-kube-api-access-s4lkq\") pod \"kuadrant-operator-catalog-w496l\" (UID: \"f763fb5c-d2d5-4832-b013-109b98dc92fb\") " pod="kuadrant-system/kuadrant-operator-catalog-w496l" Jul 07 13:35:09.139965 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:09.139935 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4lkq\" (UniqueName: \"kubernetes.io/projected/f763fb5c-d2d5-4832-b013-109b98dc92fb-kube-api-access-s4lkq\") pod \"kuadrant-operator-catalog-w496l\" (UID: \"f763fb5c-d2d5-4832-b013-109b98dc92fb\") " pod="kuadrant-system/kuadrant-operator-catalog-w496l" Jul 07 13:35:09.210113 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:09.210026 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-w496l" Jul 07 13:35:09.342655 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:09.342632 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-w496l"] Jul 07 13:35:09.350452 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:09.350424 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f5789" event={"ID":"e7bafb5c-cc08-4291-aff4-4f9989a352b6","Type":"ContainerStarted","Data":"53e606459f7c2e60ddb40cedd5e3644e3109a548c4831ae030600edad5692180"} Jul 07 13:35:09.388009 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:35:09.387976 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf763fb5c_d2d5_4832_b013_109b98dc92fb.slice/crio-0b52c375952cf30a12ee217da92e70dcb52b39d7c64f59bec564b211d95852b2 WatchSource:0}: Error finding container 0b52c375952cf30a12ee217da92e70dcb52b39d7c64f59bec564b211d95852b2: Status 404 returned error can't find the container with id 0b52c375952cf30a12ee217da92e70dcb52b39d7c64f59bec564b211d95852b2 Jul 07 13:35:10.356918 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:10.356882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-w496l" event={"ID":"f763fb5c-d2d5-4832-b013-109b98dc92fb","Type":"ContainerStarted","Data":"0b52c375952cf30a12ee217da92e70dcb52b39d7c64f59bec564b211d95852b2"} Jul 07 13:35:11.362144 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:11.362109 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-w496l" event={"ID":"f763fb5c-d2d5-4832-b013-109b98dc92fb","Type":"ContainerStarted","Data":"fb5592d774d1fb9ffaa0d4a5d36284f16284942466a117d45a86d88f5ebfacbe"} Jul 07 13:35:11.363688 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:11.363665 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f5789" event={"ID":"e7bafb5c-cc08-4291-aff4-4f9989a352b6","Type":"ContainerStarted","Data":"4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51"} Jul 07 13:35:11.363812 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:11.363731 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-f5789" podUID="e7bafb5c-cc08-4291-aff4-4f9989a352b6" containerName="registry-server" containerID="cri-o://4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51" gracePeriod=2 Jul 07 13:35:11.387581 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:11.387542 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-w496l" podStartSLOduration=1.8091660790000001 podStartE2EDuration="3.387520443s" totalImagesPullingTime="1.578354364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:35:11.385936711 +0000 UTC m=+445.387178237" watchObservedRunningTime="2026-07-07 13:35:11.387520443 +0000 UTC m=+445.388761970" Jul 07 13:35:11.409811 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:11.407003 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-f5789" podStartSLOduration=1.241635573 podStartE2EDuration="3.406985114s" totalImagesPullingTime="2.165349541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:35:11.404286508 +0000 UTC m=+445.405528028" watchObservedRunningTime="2026-07-07 13:35:11.406985114 +0000 UTC m=+445.408226642" Jul 07 13:35:11.602297 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:11.602275 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f5789" Jul 07 13:35:11.751069 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:11.751036 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt8gk\" (UniqueName: \"kubernetes.io/projected/e7bafb5c-cc08-4291-aff4-4f9989a352b6-kube-api-access-gt8gk\") pod \"e7bafb5c-cc08-4291-aff4-4f9989a352b6\" (UID: \"e7bafb5c-cc08-4291-aff4-4f9989a352b6\") " Jul 07 13:35:11.753151 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:11.753123 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bafb5c-cc08-4291-aff4-4f9989a352b6-kube-api-access-gt8gk" (OuterVolumeSpecName: "kube-api-access-gt8gk") pod "e7bafb5c-cc08-4291-aff4-4f9989a352b6" (UID: "e7bafb5c-cc08-4291-aff4-4f9989a352b6"). InnerVolumeSpecName "kube-api-access-gt8gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:35:11.851814 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:11.851787 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8gk\" (UniqueName: \"kubernetes.io/projected/e7bafb5c-cc08-4291-aff4-4f9989a352b6-kube-api-access-gt8gk\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:35:12.369018 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:12.368970 2576 generic.go:356] "Generic (PLEG): container finished" podID="e7bafb5c-cc08-4291-aff4-4f9989a352b6" containerID="4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51" exitCode=0 Jul 07 13:35:12.369018 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:12.369018 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f5789" Jul 07 13:35:12.369499 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:12.369014 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f5789" event={"ID":"e7bafb5c-cc08-4291-aff4-4f9989a352b6","Type":"ContainerDied","Data":"4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51"} Jul 07 13:35:12.369499 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:12.369150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f5789" event={"ID":"e7bafb5c-cc08-4291-aff4-4f9989a352b6","Type":"ContainerDied","Data":"53e606459f7c2e60ddb40cedd5e3644e3109a548c4831ae030600edad5692180"} Jul 07 13:35:12.369499 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:12.369198 2576 scope.go:117] "RemoveContainer" containerID="4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51" Jul 07 13:35:12.377854 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:12.377838 2576 scope.go:117] "RemoveContainer" containerID="4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51" Jul 07 13:35:12.378104 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:35:12.378083 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51\": container with ID starting with 4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51 not found: ID does not exist" containerID="4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51" Jul 07 13:35:12.378151 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:12.378112 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51"} err="failed to get container status \"4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51\": rpc error: code = NotFound desc = could not find container \"4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51\": container with ID starting with 4115206eb9ea4262dd3ce0a4d22bf94ca8dfeded1d4b0119834073e23827fc51 not found: ID does not exist" Jul 07 13:35:12.405846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:12.405828 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f5789"] Jul 07 13:35:12.411583 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:12.411564 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f5789"] Jul 07 13:35:12.589849 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:12.589817 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7bafb5c-cc08-4291-aff4-4f9989a352b6" path="/var/lib/kubelet/pods/e7bafb5c-cc08-4291-aff4-4f9989a352b6/volumes" Jul 07 13:35:19.211094 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:19.211057 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-w496l" Jul 07 13:35:19.211582 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:19.211110 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-w496l" Jul 07 13:35:19.232699 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:19.232673 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-w496l" Jul 07 13:35:19.418673 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:19.418647 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-w496l" Jul 07 13:35:40.773623 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.773588 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67"] Jul 07 13:35:40.774102 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.773877 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7bafb5c-cc08-4291-aff4-4f9989a352b6" containerName="registry-server" Jul 07 13:35:40.774102 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.773888 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7bafb5c-cc08-4291-aff4-4f9989a352b6" containerName="registry-server" Jul 07 13:35:40.776551 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.775045 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7bafb5c-cc08-4291-aff4-4f9989a352b6" containerName="registry-server" Jul 07 13:35:40.777943 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.777918 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67" Jul 07 13:35:40.781752 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.781722 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8mdzk\"" Jul 07 13:35:40.781752 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.781727 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 07 13:35:40.787155 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.787135 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67"] Jul 07 13:35:40.869059 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.869037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f65zr\" (UniqueName: \"kubernetes.io/projected/d10f6a7b-152e-4a79-b1fb-0095c9a2b3c0-kube-api-access-f65zr\") pod \"dns-operator-controller-manager-864dd78c4c-bmq67\" (UID: \"d10f6a7b-152e-4a79-b1fb-0095c9a2b3c0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67" Jul 07 13:35:40.969513 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.969485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f65zr\" (UniqueName: \"kubernetes.io/projected/d10f6a7b-152e-4a79-b1fb-0095c9a2b3c0-kube-api-access-f65zr\") pod \"dns-operator-controller-manager-864dd78c4c-bmq67\" (UID: \"d10f6a7b-152e-4a79-b1fb-0095c9a2b3c0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67" Jul 07 13:35:40.986982 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:40.986955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f65zr\" (UniqueName: \"kubernetes.io/projected/d10f6a7b-152e-4a79-b1fb-0095c9a2b3c0-kube-api-access-f65zr\") pod \"dns-operator-controller-manager-864dd78c4c-bmq67\" (UID: \"d10f6a7b-152e-4a79-b1fb-0095c9a2b3c0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67" Jul 07 13:35:41.089739 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:41.089675 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67" Jul 07 13:35:41.269455 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:41.269434 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67"] Jul 07 13:35:41.271356 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:35:41.271323 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10f6a7b_152e_4a79_b1fb_0095c9a2b3c0.slice/crio-becb320222274f70c7ef836bb16893ae7fbf793d92078b6eb99b183a548a05ac WatchSource:0}: Error finding container becb320222274f70c7ef836bb16893ae7fbf793d92078b6eb99b183a548a05ac: Status 404 returned error can't find the container with id becb320222274f70c7ef836bb16893ae7fbf793d92078b6eb99b183a548a05ac Jul 07 13:35:41.483982 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:41.483950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67" event={"ID":"d10f6a7b-152e-4a79-b1fb-0095c9a2b3c0","Type":"ContainerStarted","Data":"becb320222274f70c7ef836bb16893ae7fbf793d92078b6eb99b183a548a05ac"} Jul 07 13:35:43.494197 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:43.494145 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67" event={"ID":"d10f6a7b-152e-4a79-b1fb-0095c9a2b3c0","Type":"ContainerStarted","Data":"3e06ba484b97e364fbcd71b6946f329d7f018553dce23bb3b96ce07b32b0846f"} Jul 07 13:35:43.494542 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:43.494254 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67" Jul 07 13:35:43.516351 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:43.516302 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67" podStartSLOduration=1.486873076 podStartE2EDuration="3.516287778s" totalImagesPullingTime="2.029414702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:35:43.514938127 +0000 UTC m=+477.516179654" watchObservedRunningTime="2026-07-07 13:35:43.516287778 +0000 UTC m=+477.517529304" Jul 07 13:35:46.084023 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.083989 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw"] Jul 07 13:35:46.086372 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.086356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:35:46.089788 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.089771 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5rnfw\"" Jul 07 13:35:46.104095 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.104074 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw"] Jul 07 13:35:46.214839 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.214814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8swl\" (UniqueName: \"kubernetes.io/projected/2482da03-f7c1-40ca-bc8c-f60843cfb55b-kube-api-access-t8swl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" (UID: \"2482da03-f7c1-40ca-bc8c-f60843cfb55b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:35:46.215026 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.214863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2482da03-f7c1-40ca-bc8c-f60843cfb55b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" (UID: \"2482da03-f7c1-40ca-bc8c-f60843cfb55b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:35:46.315678 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.315642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8swl\" (UniqueName: \"kubernetes.io/projected/2482da03-f7c1-40ca-bc8c-f60843cfb55b-kube-api-access-t8swl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" (UID: \"2482da03-f7c1-40ca-bc8c-f60843cfb55b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:35:46.315796 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.315696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2482da03-f7c1-40ca-bc8c-f60843cfb55b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" (UID: \"2482da03-f7c1-40ca-bc8c-f60843cfb55b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:35:46.316026 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.316010 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2482da03-f7c1-40ca-bc8c-f60843cfb55b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" (UID: \"2482da03-f7c1-40ca-bc8c-f60843cfb55b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:35:46.333051 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.333034 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8swl\" (UniqueName: \"kubernetes.io/projected/2482da03-f7c1-40ca-bc8c-f60843cfb55b-kube-api-access-t8swl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" (UID: \"2482da03-f7c1-40ca-bc8c-f60843cfb55b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:35:46.398918 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.398872 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5rnfw\"" Jul 07 13:35:46.406511 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.406490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:35:46.536952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:46.536924 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw"] Jul 07 13:35:46.539223 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:35:46.539200 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2482da03_f7c1_40ca_bc8c_f60843cfb55b.slice/crio-26762828b275bccfb3113a04c4152b8dc38e1321c77f968ca2a36cf130a428fc WatchSource:0}: Error finding container 26762828b275bccfb3113a04c4152b8dc38e1321c77f968ca2a36cf130a428fc: Status 404 returned error can't find the container with id 26762828b275bccfb3113a04c4152b8dc38e1321c77f968ca2a36cf130a428fc Jul 07 13:35:47.514230 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:47.514193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" event={"ID":"2482da03-f7c1-40ca-bc8c-f60843cfb55b","Type":"ContainerStarted","Data":"26762828b275bccfb3113a04c4152b8dc38e1321c77f968ca2a36cf130a428fc"} Jul 07 13:35:49.173623 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:49.173589 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2"] Jul 07 13:35:49.176115 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:49.176100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:35:49.179438 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:49.179420 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hbkj9\"" Jul 07 13:35:49.190690 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:49.190671 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2"] Jul 07 13:35:49.239385 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:49.239361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrrx7\" (UniqueName: \"kubernetes.io/projected/1a423efb-c7b5-40da-8697-72a76fb21e8a-kube-api-access-zrrx7\") pod \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" (UID: \"1a423efb-c7b5-40da-8697-72a76fb21e8a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:35:49.340464 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:49.340430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrrx7\" (UniqueName: \"kubernetes.io/projected/1a423efb-c7b5-40da-8697-72a76fb21e8a-kube-api-access-zrrx7\") pod \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" (UID: \"1a423efb-c7b5-40da-8697-72a76fb21e8a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:35:49.352588 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:49.352557 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrrx7\" (UniqueName: \"kubernetes.io/projected/1a423efb-c7b5-40da-8697-72a76fb21e8a-kube-api-access-zrrx7\") pod \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" (UID: \"1a423efb-c7b5-40da-8697-72a76fb21e8a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:35:49.486365 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:49.486333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:35:49.631921 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:49.631888 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2"] Jul 07 13:35:49.633817 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:35:49.633788 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a423efb_c7b5_40da_8697_72a76fb21e8a.slice/crio-1ce595d6f84d6c61a17b764edf11fae7314676303f4bec755169a2b04c5ca0b3 WatchSource:0}: Error finding container 1ce595d6f84d6c61a17b764edf11fae7314676303f4bec755169a2b04c5ca0b3: Status 404 returned error can't find the container with id 1ce595d6f84d6c61a17b764edf11fae7314676303f4bec755169a2b04c5ca0b3 Jul 07 13:35:50.529035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:50.528997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" event={"ID":"1a423efb-c7b5-40da-8697-72a76fb21e8a","Type":"ContainerStarted","Data":"1ce595d6f84d6c61a17b764edf11fae7314676303f4bec755169a2b04c5ca0b3"} Jul 07 13:35:53.544942 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:53.544904 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" event={"ID":"2482da03-f7c1-40ca-bc8c-f60843cfb55b","Type":"ContainerStarted","Data":"14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d"} Jul 07 13:35:53.545438 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:53.545010 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:35:53.546426 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:53.546405 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" event={"ID":"1a423efb-c7b5-40da-8697-72a76fb21e8a","Type":"ContainerStarted","Data":"a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0"} Jul 07 13:35:53.546556 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:53.546541 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:35:53.597691 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:53.597640 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" podStartSLOduration=1.587124062 podStartE2EDuration="4.59762572s" totalImagesPullingTime="3.010501658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:35:53.596969158 +0000 UTC m=+487.598210687" watchObservedRunningTime="2026-07-07 13:35:53.59762572 +0000 UTC m=+487.598867247" Jul 07 13:35:53.598139 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:53.598118 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" podStartSLOduration=1.453508435 podStartE2EDuration="7.598111958s" totalImagesPullingTime="6.144603523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:35:53.576368799 +0000 UTC m=+487.577610335" watchObservedRunningTime="2026-07-07 13:35:53.598111958 +0000 UTC m=+487.599353484" Jul 07 13:35:54.502922 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:54.502894 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmq67" Jul 07 13:35:55.443532 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.443499 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn"] Jul 07 13:35:55.446002 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.445980 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.449449 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.449433 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qcjrm\"" Jul 07 13:35:55.449957 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.449941 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 07 13:35:55.450209 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.450193 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 07 13:35:55.458044 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.458021 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn"] Jul 07 13:35:55.490858 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.490832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6873865-58c6-4ec2-83d8-6e84cd7df9cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nj7zn\" (UID: \"e6873865-58c6-4ec2-83d8-6e84cd7df9cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.490939 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.490865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6873865-58c6-4ec2-83d8-6e84cd7df9cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nj7zn\" (UID: \"e6873865-58c6-4ec2-83d8-6e84cd7df9cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.490939 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.490897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrlz\" (UniqueName: \"kubernetes.io/projected/e6873865-58c6-4ec2-83d8-6e84cd7df9cc-kube-api-access-5hrlz\") pod \"kuadrant-console-plugin-66b5774985-nj7zn\" (UID: \"e6873865-58c6-4ec2-83d8-6e84cd7df9cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.591716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.591688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6873865-58c6-4ec2-83d8-6e84cd7df9cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nj7zn\" (UID: \"e6873865-58c6-4ec2-83d8-6e84cd7df9cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.591876 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.591722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6873865-58c6-4ec2-83d8-6e84cd7df9cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nj7zn\" (UID: \"e6873865-58c6-4ec2-83d8-6e84cd7df9cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.591876 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.591747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrlz\" (UniqueName: \"kubernetes.io/projected/e6873865-58c6-4ec2-83d8-6e84cd7df9cc-kube-api-access-5hrlz\") pod \"kuadrant-console-plugin-66b5774985-nj7zn\" (UID: \"e6873865-58c6-4ec2-83d8-6e84cd7df9cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.592407 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.592385 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6873865-58c6-4ec2-83d8-6e84cd7df9cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nj7zn\" (UID: \"e6873865-58c6-4ec2-83d8-6e84cd7df9cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.594014 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.593994 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6873865-58c6-4ec2-83d8-6e84cd7df9cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nj7zn\" (UID: \"e6873865-58c6-4ec2-83d8-6e84cd7df9cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.604009 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.603991 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrlz\" (UniqueName: \"kubernetes.io/projected/e6873865-58c6-4ec2-83d8-6e84cd7df9cc-kube-api-access-5hrlz\") pod \"kuadrant-console-plugin-66b5774985-nj7zn\" (UID: \"e6873865-58c6-4ec2-83d8-6e84cd7df9cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.755046 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.755022 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" Jul 07 13:35:55.917588 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:55.917563 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn"] Jul 07 13:35:55.920026 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:35:55.919999 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6873865_58c6_4ec2_83d8_6e84cd7df9cc.slice/crio-74f7fa03ee968f994465f5297ab07b42de6f4369288ed21b8ef2ab926f107f75 WatchSource:0}: Error finding container 74f7fa03ee968f994465f5297ab07b42de6f4369288ed21b8ef2ab926f107f75: Status 404 returned error can't find the container with id 74f7fa03ee968f994465f5297ab07b42de6f4369288ed21b8ef2ab926f107f75 Jul 07 13:35:56.559675 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:35:56.559641 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" event={"ID":"e6873865-58c6-4ec2-83d8-6e84cd7df9cc","Type":"ContainerStarted","Data":"74f7fa03ee968f994465f5297ab07b42de6f4369288ed21b8ef2ab926f107f75"} Jul 07 13:36:04.553397 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:04.553361 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:04.553829 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:04.553441 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:06.338193 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.338131 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt"] Jul 07 13:36:06.351467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.351435 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:06.354283 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.354256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt"] Jul 07 13:36:06.484211 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.484143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/775ce9cc-8733-485e-b298-e58a5f2c2346-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g9tt\" (UID: \"775ce9cc-8733-485e-b298-e58a5f2c2346\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:06.484400 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.484274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmp2m\" (UniqueName: \"kubernetes.io/projected/775ce9cc-8733-485e-b298-e58a5f2c2346-kube-api-access-bmp2m\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g9tt\" (UID: \"775ce9cc-8733-485e-b298-e58a5f2c2346\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:06.501265 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.501237 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt"] Jul 07 13:36:06.501509 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:06.501480 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-bmp2m], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" podUID="775ce9cc-8733-485e-b298-e58a5f2c2346" Jul 07 13:36:06.516860 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.516837 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt"] Jul 07 13:36:06.541258 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.541230 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw"] Jul 07 13:36:06.541463 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.541439 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" containerName="manager" containerID="cri-o://14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d" gracePeriod=2 Jul 07 13:36:06.542952 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.542904 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c"] Jul 07 13:36:06.548126 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.548106 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" Jul 07 13:36:06.554229 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.554153 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw"] Jul 07 13:36:06.555640 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:06.555612 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:06.567260 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.567142 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2"] Jul 07 13:36:06.567480 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.567355 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" containerName="manager" containerID="cri-o://a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0" gracePeriod=2 Jul 07 13:36:06.569054 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.569018 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c"] Jul 07 13:36:06.578648 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.578563 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2"] Jul 07 13:36:06.588234 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.586234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/775ce9cc-8733-485e-b298-e58a5f2c2346-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g9tt\" (UID: \"775ce9cc-8733-485e-b298-e58a5f2c2346\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:06.588234 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.587966 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/775ce9cc-8733-485e-b298-e58a5f2c2346-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g9tt\" (UID: \"775ce9cc-8733-485e-b298-e58a5f2c2346\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:06.588234 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.588031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmp2m\" (UniqueName: \"kubernetes.io/projected/775ce9cc-8733-485e-b298-e58a5f2c2346-kube-api-access-bmp2m\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g9tt\" (UID: \"775ce9cc-8733-485e-b298-e58a5f2c2346\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:06.601417 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:06.601394 2576 projected.go:196] Error preparing data for projected volume kube-api-access-bmp2m for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt: failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-6g9tt" not found Jul 07 13:36:06.601743 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:06.601464 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775ce9cc-8733-485e-b298-e58a5f2c2346-kube-api-access-bmp2m podName:775ce9cc-8733-485e-b298-e58a5f2c2346 nodeName:}" failed. No retries permitted until 2026-07-07 13:36:07.101445402 +0000 UTC m=+501.102686907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bmp2m" (UniqueName: "kubernetes.io/projected/775ce9cc-8733-485e-b298-e58a5f2c2346-kube-api-access-bmp2m") pod "kuadrant-operator-controller-manager-55467d9f7-6g9tt" (UID: "775ce9cc-8733-485e-b298-e58a5f2c2346") : failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-6g9tt" not found Jul 07 13:36:06.603951 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.603926 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm"] Jul 07 13:36:06.604345 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.604326 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" containerName="manager" Jul 07 13:36:06.604345 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.604345 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" containerName="manager" Jul 07 13:36:06.604490 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.604364 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" containerName="manager" Jul 07 13:36:06.604490 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.604372 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" containerName="manager" Jul 07 13:36:06.604490 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.604464 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" containerName="manager" Jul 07 13:36:06.604490 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.604475 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" containerName="manager" Jul 07 13:36:06.606927 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.606725 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8"] Jul 07 13:36:06.606927 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:06.606743 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:06.607265 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.607245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:06.610979 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:06.610954 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:06.611968 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.611946 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:06.617337 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.617317 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm"] Jul 07 13:36:06.617442 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.617409 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8" Jul 07 13:36:06.626279 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.626256 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c"] Jul 07 13:36:06.627866 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.627843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8"] Jul 07 13:36:06.664783 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:06.664751 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-qnn8g], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" podUID="8976f8e0-74e6-4c2f-8374-adfdbd1f06c7" Jul 07 13:36:06.676071 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.676051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:06.680353 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:06.680317 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-6g9tt\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="775ce9cc-8733-485e-b298-e58a5f2c2346" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:06.689068 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.689046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6ftn\" (UniqueName: \"kubernetes.io/projected/6720d4e8-917a-4f91-a721-085bd04a35fa-kube-api-access-w6ftn\") pod \"limitador-operator-controller-manager-859d7b78bd-l8bx8\" (UID: \"6720d4e8-917a-4f91-a721-085bd04a35fa\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8" Jul 07 13:36:06.689177 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.689099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6c8c\" (UID: \"8976f8e0-74e6-4c2f-8374-adfdbd1f06c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" Jul 07 13:36:06.689237 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.689217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbkqg\" (UniqueName: \"kubernetes.io/projected/eb71e41b-c3df-401a-9a11-06bd2005a2cb-kube-api-access-bbkqg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-28sdm\" (UID: \"eb71e41b-c3df-401a-9a11-06bd2005a2cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:06.689278 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.689261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb71e41b-c3df-401a-9a11-06bd2005a2cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-28sdm\" (UID: \"eb71e41b-c3df-401a-9a11-06bd2005a2cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:06.689349 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.689330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnn8g\" (UniqueName: \"kubernetes.io/projected/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-kube-api-access-qnn8g\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6c8c\" (UID: \"8976f8e0-74e6-4c2f-8374-adfdbd1f06c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" Jul 07 13:36:06.789829 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.789792 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/775ce9cc-8733-485e-b298-e58a5f2c2346-extensions-socket-volume\") pod \"775ce9cc-8733-485e-b298-e58a5f2c2346\" (UID: \"775ce9cc-8733-485e-b298-e58a5f2c2346\") " Jul 07 13:36:06.789991 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.789941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ftn\" (UniqueName: \"kubernetes.io/projected/6720d4e8-917a-4f91-a721-085bd04a35fa-kube-api-access-w6ftn\") pod \"limitador-operator-controller-manager-859d7b78bd-l8bx8\" (UID: \"6720d4e8-917a-4f91-a721-085bd04a35fa\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8" Jul 07 13:36:06.790051 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.789993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6c8c\" (UID: \"8976f8e0-74e6-4c2f-8374-adfdbd1f06c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" Jul 07 13:36:06.790051 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.790014 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775ce9cc-8733-485e-b298-e58a5f2c2346-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "775ce9cc-8733-485e-b298-e58a5f2c2346" (UID: "775ce9cc-8733-485e-b298-e58a5f2c2346"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:36:06.790153 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.790069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbkqg\" (UniqueName: \"kubernetes.io/projected/eb71e41b-c3df-401a-9a11-06bd2005a2cb-kube-api-access-bbkqg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-28sdm\" (UID: \"eb71e41b-c3df-401a-9a11-06bd2005a2cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:06.790153 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.790106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb71e41b-c3df-401a-9a11-06bd2005a2cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-28sdm\" (UID: \"eb71e41b-c3df-401a-9a11-06bd2005a2cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:06.790302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.790160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnn8g\" (UniqueName: \"kubernetes.io/projected/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-kube-api-access-qnn8g\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6c8c\" (UID: \"8976f8e0-74e6-4c2f-8374-adfdbd1f06c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" Jul 07 13:36:06.790302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.790272 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/775ce9cc-8733-485e-b298-e58a5f2c2346-extensions-socket-volume\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:36:06.790302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.790288 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmp2m\" (UniqueName: \"kubernetes.io/projected/775ce9cc-8733-485e-b298-e58a5f2c2346-kube-api-access-bmp2m\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:36:06.790443 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.790362 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6c8c\" (UID: \"8976f8e0-74e6-4c2f-8374-adfdbd1f06c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" Jul 07 13:36:06.790582 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.790559 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb71e41b-c3df-401a-9a11-06bd2005a2cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-28sdm\" (UID: \"eb71e41b-c3df-401a-9a11-06bd2005a2cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:06.809272 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.809247 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnn8g\" (UniqueName: \"kubernetes.io/projected/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-kube-api-access-qnn8g\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6c8c\" (UID: \"8976f8e0-74e6-4c2f-8374-adfdbd1f06c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" Jul 07 13:36:06.809404 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.809325 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6ftn\" (UniqueName: \"kubernetes.io/projected/6720d4e8-917a-4f91-a721-085bd04a35fa-kube-api-access-w6ftn\") pod \"limitador-operator-controller-manager-859d7b78bd-l8bx8\" (UID: \"6720d4e8-917a-4f91-a721-085bd04a35fa\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8" Jul 07 13:36:06.809826 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.809801 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbkqg\" (UniqueName: \"kubernetes.io/projected/eb71e41b-c3df-401a-9a11-06bd2005a2cb-kube-api-access-bbkqg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-28sdm\" (UID: \"eb71e41b-c3df-401a-9a11-06bd2005a2cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:06.973379 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.973293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:06.985439 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:06.985412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8" Jul 07 13:36:07.617015 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:07.616986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" Jul 07 13:36:07.617015 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:07.617021 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:07.620213 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:07.620176 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-6g9tt\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="775ce9cc-8733-485e-b298-e58a5f2c2346" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:07.632201 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:07.632150 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" Jul 07 13:36:07.670889 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:07.670853 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-6g9tt\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="775ce9cc-8733-485e-b298-e58a5f2c2346" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g9tt" Jul 07 13:36:07.699175 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:07.699149 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnn8g\" (UniqueName: \"kubernetes.io/projected/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-kube-api-access-qnn8g\") pod \"8976f8e0-74e6-4c2f-8374-adfdbd1f06c7\" (UID: \"8976f8e0-74e6-4c2f-8374-adfdbd1f06c7\") " Jul 07 13:36:07.699308 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:07.699218 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-extensions-socket-volume\") pod \"8976f8e0-74e6-4c2f-8374-adfdbd1f06c7\" (UID: \"8976f8e0-74e6-4c2f-8374-adfdbd1f06c7\") " Jul 07 13:36:07.699540 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:07.699508 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8976f8e0-74e6-4c2f-8374-adfdbd1f06c7" (UID: "8976f8e0-74e6-4c2f-8374-adfdbd1f06c7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:36:07.701574 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:07.701545 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-kube-api-access-qnn8g" (OuterVolumeSpecName: "kube-api-access-qnn8g") pod "8976f8e0-74e6-4c2f-8374-adfdbd1f06c7" (UID: "8976f8e0-74e6-4c2f-8374-adfdbd1f06c7"). InnerVolumeSpecName "kube-api-access-qnn8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:36:07.800010 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:07.799972 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnn8g\" (UniqueName: \"kubernetes.io/projected/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-kube-api-access-qnn8g\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:36:07.800010 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:07.799999 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7-extensions-socket-volume\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:36:08.590115 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:08.590083 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775ce9cc-8733-485e-b298-e58a5f2c2346" path="/var/lib/kubelet/pods/775ce9cc-8733-485e-b298-e58a5f2c2346/volumes" Jul 07 13:36:08.621336 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:08.621305 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c" Jul 07 13:36:08.668396 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:08.668343 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c"] Jul 07 13:36:08.679594 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:08.679547 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6c8c"] Jul 07 13:36:10.590582 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:10.590553 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8976f8e0-74e6-4c2f-8374-adfdbd1f06c7" path="/var/lib/kubelet/pods/8976f8e0-74e6-4c2f-8374-adfdbd1f06c7/volumes" Jul 07 13:36:18.662457 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.662390 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm"] Jul 07 13:36:18.664747 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:36:18.664695 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb71e41b_c3df_401a_9a11_06bd2005a2cb.slice/crio-e7d76a844348f370c9253de615dd741e7c6beabc0faf5afadf04a70a51e43044 WatchSource:0}: Error finding container e7d76a844348f370c9253de615dd741e7c6beabc0faf5afadf04a70a51e43044: Status 404 returned error can't find the container with id e7d76a844348f370c9253de615dd741e7c6beabc0faf5afadf04a70a51e43044 Jul 07 13:36:18.664858 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.664752 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:18.666099 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.666077 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:18.668247 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.668222 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:18.668684 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.668661 2576 generic.go:356] "Generic (PLEG): container finished" podID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" containerID="14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d" exitCode=0 Jul 07 13:36:18.668772 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.668690 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:18.668834 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.668771 2576 scope.go:117] "RemoveContainer" containerID="14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d" Jul 07 13:36:18.669993 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.669972 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a423efb-c7b5-40da-8697-72a76fb21e8a" containerID="a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0" exitCode=0 Jul 07 13:36:18.670075 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.669999 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:18.671400 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.671368 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:18.673742 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.673719 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:18.677018 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.676996 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:18.679283 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.679259 2576 scope.go:117] "RemoveContainer" containerID="14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d" Jul 07 13:36:18.679575 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.679556 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d\": container with ID starting with 14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d not found: ID does not exist" containerID="14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d" Jul 07 13:36:18.679662 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.679588 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d"} err="failed to get container status \"14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d\": rpc error: code = NotFound desc = could not find container \"14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d\": container with ID starting with 14fce1859a9f40c1a199c2d227b18227511fb539778c1675bae79086879e119d not found: ID does not exist" Jul 07 13:36:18.679662 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.679610 2576 scope.go:117] "RemoveContainer" containerID="a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0" Jul 07 13:36:18.679968 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.679939 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:18.687661 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.687646 2576 scope.go:117] "RemoveContainer" containerID="a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0" Jul 07 13:36:18.688478 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.688451 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0\": container with ID starting with a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0 not found: ID does not exist" containerID="a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0" Jul 07 13:36:18.688577 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.688485 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0"} err="failed to get container status \"a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0\": rpc error: code = NotFound desc = could not find container \"a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0\": container with ID starting with a03d38bf86522c259c71129f67f190ad0b3db0722c30537ef87c28fb166bd1c0 not found: ID does not exist" Jul 07 13:36:18.689261 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.689155 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8"] Jul 07 13:36:18.698292 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:36:18.698266 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6720d4e8_917a_4f91_a721_085bd04a35fa.slice/crio-14c3dd975e504a8459e5d4e16b93543abaeeaa5bcf7dcfce2514d9cb529aae58 WatchSource:0}: Error finding container 14c3dd975e504a8459e5d4e16b93543abaeeaa5bcf7dcfce2514d9cb529aae58: Status 404 returned error can't find the container with id 14c3dd975e504a8459e5d4e16b93543abaeeaa5bcf7dcfce2514d9cb529aae58 Jul 07 13:36:18.800447 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.800386 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2482da03-f7c1-40ca-bc8c-f60843cfb55b-extensions-socket-volume\") pod \"2482da03-f7c1-40ca-bc8c-f60843cfb55b\" (UID: \"2482da03-f7c1-40ca-bc8c-f60843cfb55b\") " Jul 07 13:36:18.800447 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.800430 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8swl\" (UniqueName: \"kubernetes.io/projected/2482da03-f7c1-40ca-bc8c-f60843cfb55b-kube-api-access-t8swl\") pod \"2482da03-f7c1-40ca-bc8c-f60843cfb55b\" (UID: \"2482da03-f7c1-40ca-bc8c-f60843cfb55b\") " Jul 07 13:36:18.800609 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.800472 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrrx7\" (UniqueName: \"kubernetes.io/projected/1a423efb-c7b5-40da-8697-72a76fb21e8a-kube-api-access-zrrx7\") pod \"1a423efb-c7b5-40da-8697-72a76fb21e8a\" (UID: \"1a423efb-c7b5-40da-8697-72a76fb21e8a\") " Jul 07 13:36:18.800919 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.800890 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2482da03-f7c1-40ca-bc8c-f60843cfb55b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2482da03-f7c1-40ca-bc8c-f60843cfb55b" (UID: "2482da03-f7c1-40ca-bc8c-f60843cfb55b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:36:18.802424 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.802402 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a423efb-c7b5-40da-8697-72a76fb21e8a-kube-api-access-zrrx7" (OuterVolumeSpecName: "kube-api-access-zrrx7") pod "1a423efb-c7b5-40da-8697-72a76fb21e8a" (UID: "1a423efb-c7b5-40da-8697-72a76fb21e8a"). InnerVolumeSpecName "kube-api-access-zrrx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:36:18.802490 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.802429 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2482da03-f7c1-40ca-bc8c-f60843cfb55b-kube-api-access-t8swl" (OuterVolumeSpecName: "kube-api-access-t8swl") pod "2482da03-f7c1-40ca-bc8c-f60843cfb55b" (UID: "2482da03-f7c1-40ca-bc8c-f60843cfb55b"). InnerVolumeSpecName "kube-api-access-t8swl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:36:18.901657 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.901623 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8swl\" (UniqueName: \"kubernetes.io/projected/2482da03-f7c1-40ca-bc8c-f60843cfb55b-kube-api-access-t8swl\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:36:18.901657 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.901647 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrrx7\" (UniqueName: \"kubernetes.io/projected/1a423efb-c7b5-40da-8697-72a76fb21e8a-kube-api-access-zrrx7\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:36:18.901657 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:18.901656 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2482da03-f7c1-40ca-bc8c-f60843cfb55b-extensions-socket-volume\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:36:18.980198 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.980153 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:18.982491 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.982468 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:18.986268 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.986248 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:18.989580 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:18.989564 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:19.677283 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:19.677251 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" event={"ID":"e6873865-58c6-4ec2-83d8-6e84cd7df9cc","Type":"ContainerStarted","Data":"551563ee5ad0ef9ca30ff65d55271051cc2ca7f49e44366aada977917be9eb11"} Jul 07 13:36:19.678749 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:19.678724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" event={"ID":"eb71e41b-c3df-401a-9a11-06bd2005a2cb","Type":"ContainerStarted","Data":"f5d5d1cd1a575d180d1b1620e9b65fe601e1fb90a092967bfe2d0945f7966fa6"} Jul 07 13:36:19.678876 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:19.678774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" event={"ID":"eb71e41b-c3df-401a-9a11-06bd2005a2cb","Type":"ContainerStarted","Data":"e7d76a844348f370c9253de615dd741e7c6beabc0faf5afadf04a70a51e43044"} Jul 07 13:36:19.678876 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:19.678803 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:19.681704 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:19.681681 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8" event={"ID":"6720d4e8-917a-4f91-a721-085bd04a35fa","Type":"ContainerStarted","Data":"ec1eb4626ec09c56f41ae821cf5a2ac8f7ae41be046720ef5167cf5d9e11fb29"} Jul 07 13:36:19.681837 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:19.681719 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8" event={"ID":"6720d4e8-917a-4f91-a721-085bd04a35fa","Type":"ContainerStarted","Data":"14c3dd975e504a8459e5d4e16b93543abaeeaa5bcf7dcfce2514d9cb529aae58"} Jul 07 13:36:19.681837 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:19.681798 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8" Jul 07 13:36:19.699145 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:19.699098 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj7zn" podStartSLOduration=2.044211389 podStartE2EDuration="24.699085617s" totalImagesPullingTime="22.654874228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:36:19.698137749 +0000 UTC m=+513.699379276" watchObservedRunningTime="2026-07-07 13:36:19.699085617 +0000 UTC m=+513.700327144" Jul 07 13:36:19.700923 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:19.700903 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:19.703298 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:19.703281 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:19.705783 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:19.705759 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddklw\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddklw" Jul 07 13:36:19.708018 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:19.707994 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9zrj2\" is forbidden: User \"system:node:ip-10-0-132-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-236.ec2.internal' and this object" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9zrj2" Jul 07 13:36:19.730456 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:19.730420 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" podStartSLOduration=13.730407233 podStartE2EDuration="13.730407233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:36:19.729227193 +0000 UTC m=+513.730468722" watchObservedRunningTime="2026-07-07 13:36:19.730407233 +0000 UTC m=+513.731648759" Jul 07 13:36:19.750923 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:19.750853 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8" podStartSLOduration=13.750842052 podStartE2EDuration="13.750842052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:36:19.749700758 +0000 UTC m=+513.750942285" watchObservedRunningTime="2026-07-07 13:36:19.750842052 +0000 UTC m=+513.752083579" Jul 07 13:36:20.589787 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:20.589751 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a423efb-c7b5-40da-8697-72a76fb21e8a" path="/var/lib/kubelet/pods/1a423efb-c7b5-40da-8697-72a76fb21e8a/volumes" Jul 07 13:36:20.590093 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:20.590081 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2482da03-f7c1-40ca-bc8c-f60843cfb55b" path="/var/lib/kubelet/pods/2482da03-f7c1-40ca-bc8c-f60843cfb55b/volumes" Jul 07 13:36:24.466536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.466506 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm"] Jul 07 13:36:24.467011 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.466715 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" podUID="eb71e41b-c3df-401a-9a11-06bd2005a2cb" containerName="manager" containerID="cri-o://f5d5d1cd1a575d180d1b1620e9b65fe601e1fb90a092967bfe2d0945f7966fa6" gracePeriod=10 Jul 07 13:36:24.468543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.468519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:24.706816 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.706785 2576 generic.go:356] "Generic (PLEG): container finished" podID="eb71e41b-c3df-401a-9a11-06bd2005a2cb" containerID="f5d5d1cd1a575d180d1b1620e9b65fe601e1fb90a092967bfe2d0945f7966fa6" exitCode=0 Jul 07 13:36:24.706966 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.706836 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" event={"ID":"eb71e41b-c3df-401a-9a11-06bd2005a2cb","Type":"ContainerDied","Data":"f5d5d1cd1a575d180d1b1620e9b65fe601e1fb90a092967bfe2d0945f7966fa6"} Jul 07 13:36:24.748673 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.748654 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:24.849034 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.848994 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb71e41b-c3df-401a-9a11-06bd2005a2cb-extensions-socket-volume\") pod \"eb71e41b-c3df-401a-9a11-06bd2005a2cb\" (UID: \"eb71e41b-c3df-401a-9a11-06bd2005a2cb\") " Jul 07 13:36:24.849209 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.849096 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbkqg\" (UniqueName: \"kubernetes.io/projected/eb71e41b-c3df-401a-9a11-06bd2005a2cb-kube-api-access-bbkqg\") pod \"eb71e41b-c3df-401a-9a11-06bd2005a2cb\" (UID: \"eb71e41b-c3df-401a-9a11-06bd2005a2cb\") " Jul 07 13:36:24.849473 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.849443 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb71e41b-c3df-401a-9a11-06bd2005a2cb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eb71e41b-c3df-401a-9a11-06bd2005a2cb" (UID: "eb71e41b-c3df-401a-9a11-06bd2005a2cb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:36:24.851821 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.851792 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb71e41b-c3df-401a-9a11-06bd2005a2cb-kube-api-access-bbkqg" (OuterVolumeSpecName: "kube-api-access-bbkqg") pod "eb71e41b-c3df-401a-9a11-06bd2005a2cb" (UID: "eb71e41b-c3df-401a-9a11-06bd2005a2cb"). InnerVolumeSpecName "kube-api-access-bbkqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:36:24.949906 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.949874 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb71e41b-c3df-401a-9a11-06bd2005a2cb-extensions-socket-volume\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:36:24.949906 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:24.949903 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbkqg\" (UniqueName: \"kubernetes.io/projected/eb71e41b-c3df-401a-9a11-06bd2005a2cb-kube-api-access-bbkqg\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:36:25.712961 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:25.712937 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" Jul 07 13:36:25.713411 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:25.712934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm" event={"ID":"eb71e41b-c3df-401a-9a11-06bd2005a2cb","Type":"ContainerDied","Data":"e7d76a844348f370c9253de615dd741e7c6beabc0faf5afadf04a70a51e43044"} Jul 07 13:36:25.713411 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:25.713078 2576 scope.go:117] "RemoveContainer" containerID="f5d5d1cd1a575d180d1b1620e9b65fe601e1fb90a092967bfe2d0945f7966fa6" Jul 07 13:36:25.736536 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:25.736509 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm"] Jul 07 13:36:25.741432 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:25.741411 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-28sdm"] Jul 07 13:36:26.589543 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:26.589512 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb71e41b-c3df-401a-9a11-06bd2005a2cb" path="/var/lib/kubelet/pods/eb71e41b-c3df-401a-9a11-06bd2005a2cb/volumes" Jul 07 13:36:30.688302 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:30.688273 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bx8" Jul 07 13:36:40.894389 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:40.894309 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6"] Jul 07 13:36:40.894810 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:40.894755 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb71e41b-c3df-401a-9a11-06bd2005a2cb" containerName="manager" Jul 07 13:36:40.894810 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:40.894768 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb71e41b-c3df-401a-9a11-06bd2005a2cb" containerName="manager" Jul 07 13:36:40.894886 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:40.894828 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb71e41b-c3df-401a-9a11-06bd2005a2cb" containerName="manager" Jul 07 13:36:40.962333 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:40.962305 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6"] Jul 07 13:36:40.962507 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:40.962443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:40.967116 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:40.967093 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qzqkp\"" Jul 07 13:36:41.083467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.083432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcca19d0-38dd-4c17-b2a6-c9520788d994-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.083467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.083469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.083689 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.083538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpp4l\" (UniqueName: \"kubernetes.io/projected/dcca19d0-38dd-4c17-b2a6-c9520788d994-kube-api-access-gpp4l\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.083689 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.083562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.083689 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.083596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.083689 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.083639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.083831 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.083699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.083831 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.083721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.083831 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.083739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184453 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcca19d0-38dd-4c17-b2a6-c9520788d994-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184453 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184623 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpp4l\" (UniqueName: \"kubernetes.io/projected/dcca19d0-38dd-4c17-b2a6-c9520788d994-kube-api-access-gpp4l\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184623 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184623 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184738 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184796 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184796 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184914 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184914 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184868 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.184914 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.184888 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.185124 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.185102 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.185216 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.185195 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.185285 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.185197 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcca19d0-38dd-4c17-b2a6-c9520788d994-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.187036 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.187009 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.187177 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.187097 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.199501 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.199473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpp4l\" (UniqueName: \"kubernetes.io/projected/dcca19d0-38dd-4c17-b2a6-c9520788d994-kube-api-access-gpp4l\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.199635 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.199620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcca19d0-38dd-4c17-b2a6-c9520788d994-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-4qqm6\" (UID: \"dcca19d0-38dd-4c17-b2a6-c9520788d994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.271995 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.271958 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:41.425652 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.425615 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6"] Jul 07 13:36:41.428070 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:36:41.428043 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcca19d0_38dd_4c17_b2a6_c9520788d994.slice/crio-a9930b0dbffcda1dc758922be939e47a9399c9639c44875995820a1106977999 WatchSource:0}: Error finding container a9930b0dbffcda1dc758922be939e47a9399c9639c44875995820a1106977999: Status 404 returned error can't find the container with id a9930b0dbffcda1dc758922be939e47a9399c9639c44875995820a1106977999 Jul 07 13:36:41.430208 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.430175 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:36:41.430307 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.430255 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:36:41.430366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.430303 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:36:41.793671 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.793638 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" event={"ID":"dcca19d0-38dd-4c17-b2a6-c9520788d994","Type":"ContainerStarted","Data":"0809274e623e6f8b6ff1608f769fbbd6786bc5eab4c8845ada7902987aa981c1"} Jul 07 13:36:41.793825 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.793690 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" event={"ID":"dcca19d0-38dd-4c17-b2a6-c9520788d994","Type":"ContainerStarted","Data":"a9930b0dbffcda1dc758922be939e47a9399c9639c44875995820a1106977999"} Jul 07 13:36:41.823572 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:41.823478 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" podStartSLOduration=1.8234621039999999 podStartE2EDuration="1.823462104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:36:41.822516795 +0000 UTC m=+535.823758322" watchObservedRunningTime="2026-07-07 13:36:41.823462104 +0000 UTC m=+535.824703610" Jul 07 13:36:42.272644 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:42.272611 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:42.277074 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:42.277050 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:42.798020 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:42.797989 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:42.798887 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:42.798868 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:36:43.134683 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.134614 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hhwz6"] Jul 07 13:36:43.394224 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.394137 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hhwz6"] Jul 07 13:36:43.394224 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.394191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hhwz6"] Jul 07 13:36:43.394686 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.394279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:36:43.398722 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.398705 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 13:36:43.506859 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.506826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/beaba350-7748-4ae6-8596-181d40e432c9-config-file\") pod \"limitador-limitador-64c94fd64f-hhwz6\" (UID: \"beaba350-7748-4ae6-8596-181d40e432c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:36:43.507015 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.506931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwdlc\" (UniqueName: \"kubernetes.io/projected/beaba350-7748-4ae6-8596-181d40e432c9-kube-api-access-gwdlc\") pod \"limitador-limitador-64c94fd64f-hhwz6\" (UID: \"beaba350-7748-4ae6-8596-181d40e432c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:36:43.607980 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.607927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwdlc\" (UniqueName: \"kubernetes.io/projected/beaba350-7748-4ae6-8596-181d40e432c9-kube-api-access-gwdlc\") pod \"limitador-limitador-64c94fd64f-hhwz6\" (UID: \"beaba350-7748-4ae6-8596-181d40e432c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:36:43.608145 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.607999 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/beaba350-7748-4ae6-8596-181d40e432c9-config-file\") pod \"limitador-limitador-64c94fd64f-hhwz6\" (UID: \"beaba350-7748-4ae6-8596-181d40e432c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:36:43.608639 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.608621 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/beaba350-7748-4ae6-8596-181d40e432c9-config-file\") pod \"limitador-limitador-64c94fd64f-hhwz6\" (UID: \"beaba350-7748-4ae6-8596-181d40e432c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:36:43.619627 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.619604 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwdlc\" (UniqueName: \"kubernetes.io/projected/beaba350-7748-4ae6-8596-181d40e432c9-kube-api-access-gwdlc\") pod \"limitador-limitador-64c94fd64f-hhwz6\" (UID: \"beaba350-7748-4ae6-8596-181d40e432c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:36:43.704650 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.704583 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:36:43.866609 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.866585 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hhwz6"] Jul 07 13:36:43.868996 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:36:43.868964 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeaba350_7748_4ae6_8596_181d40e432c9.slice/crio-1d55d3ab5a01ea2bf540e4afd32d90e8840693f97c61f426cda72c987e7c244f WatchSource:0}: Error finding container 1d55d3ab5a01ea2bf540e4afd32d90e8840693f97c61f426cda72c987e7c244f: Status 404 returned error can't find the container with id 1d55d3ab5a01ea2bf540e4afd32d90e8840693f97c61f426cda72c987e7c244f Jul 07 13:36:43.999092 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:43.999055 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-pcpfv"] Jul 07 13:36:44.105662 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.105633 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-pcpfv"] Jul 07 13:36:44.105846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.105737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-pcpfv" Jul 07 13:36:44.110266 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.110246 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q49v7\"" Jul 07 13:36:44.211219 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.211190 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-cmj7v"] Jul 07 13:36:44.212867 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.212837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf9rf\" (UniqueName: \"kubernetes.io/projected/08d5cb0a-5e95-41fe-a073-b0e982df4f43-kube-api-access-wf9rf\") pod \"authorino-58cd955675-pcpfv\" (UID: \"08d5cb0a-5e95-41fe-a073-b0e982df4f43\") " pod="kuadrant-system/authorino-58cd955675-pcpfv" Jul 07 13:36:44.256597 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.256524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cmj7v"] Jul 07 13:36:44.256708 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.256616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cmj7v" Jul 07 13:36:44.314018 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.313988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf9rf\" (UniqueName: \"kubernetes.io/projected/08d5cb0a-5e95-41fe-a073-b0e982df4f43-kube-api-access-wf9rf\") pod \"authorino-58cd955675-pcpfv\" (UID: \"08d5cb0a-5e95-41fe-a073-b0e982df4f43\") " pod="kuadrant-system/authorino-58cd955675-pcpfv" Jul 07 13:36:44.314147 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.314037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wf7\" (UniqueName: \"kubernetes.io/projected/041b74ba-8105-469f-8795-1fd332599663-kube-api-access-h8wf7\") pod \"authorino-75d556bd77-cmj7v\" (UID: \"041b74ba-8105-469f-8795-1fd332599663\") " pod="kuadrant-system/authorino-75d556bd77-cmj7v" Jul 07 13:36:44.328949 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.328927 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf9rf\" (UniqueName: \"kubernetes.io/projected/08d5cb0a-5e95-41fe-a073-b0e982df4f43-kube-api-access-wf9rf\") pod \"authorino-58cd955675-pcpfv\" (UID: \"08d5cb0a-5e95-41fe-a073-b0e982df4f43\") " pod="kuadrant-system/authorino-58cd955675-pcpfv" Jul 07 13:36:44.414738 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.414707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-pcpfv" Jul 07 13:36:44.415099 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.414981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wf7\" (UniqueName: \"kubernetes.io/projected/041b74ba-8105-469f-8795-1fd332599663-kube-api-access-h8wf7\") pod \"authorino-75d556bd77-cmj7v\" (UID: \"041b74ba-8105-469f-8795-1fd332599663\") " pod="kuadrant-system/authorino-75d556bd77-cmj7v" Jul 07 13:36:44.434057 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.434034 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wf7\" (UniqueName: \"kubernetes.io/projected/041b74ba-8105-469f-8795-1fd332599663-kube-api-access-h8wf7\") pod \"authorino-75d556bd77-cmj7v\" (UID: \"041b74ba-8105-469f-8795-1fd332599663\") " pod="kuadrant-system/authorino-75d556bd77-cmj7v" Jul 07 13:36:44.566232 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.566197 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cmj7v" Jul 07 13:36:44.679297 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.678802 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-pcpfv"] Jul 07 13:36:44.684497 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:36:44.684465 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08d5cb0a_5e95_41fe_a073_b0e982df4f43.slice/crio-ea47e60ddcb2cdd0458e341d555fec46e0ae65060c968cf5aa2a48f70b286718 WatchSource:0}: Error finding container ea47e60ddcb2cdd0458e341d555fec46e0ae65060c968cf5aa2a48f70b286718: Status 404 returned error can't find the container with id ea47e60ddcb2cdd0458e341d555fec46e0ae65060c968cf5aa2a48f70b286718 Jul 07 13:36:44.762965 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:36:44.762937 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod041b74ba_8105_469f_8795_1fd332599663.slice/crio-097cb7e741dc723fd2e523f4f57430f159e5a23b21d94aae507a7da9c39c5db4 WatchSource:0}: Error finding container 097cb7e741dc723fd2e523f4f57430f159e5a23b21d94aae507a7da9c39c5db4: Status 404 returned error can't find the container with id 097cb7e741dc723fd2e523f4f57430f159e5a23b21d94aae507a7da9c39c5db4 Jul 07 13:36:44.765546 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.765524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cmj7v"] Jul 07 13:36:44.809087 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.809011 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cmj7v" event={"ID":"041b74ba-8105-469f-8795-1fd332599663","Type":"ContainerStarted","Data":"097cb7e741dc723fd2e523f4f57430f159e5a23b21d94aae507a7da9c39c5db4"} Jul 07 13:36:44.810215 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.810185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" event={"ID":"beaba350-7748-4ae6-8596-181d40e432c9","Type":"ContainerStarted","Data":"1d55d3ab5a01ea2bf540e4afd32d90e8840693f97c61f426cda72c987e7c244f"} Jul 07 13:36:44.811243 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:44.811224 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-pcpfv" event={"ID":"08d5cb0a-5e95-41fe-a073-b0e982df4f43","Type":"ContainerStarted","Data":"ea47e60ddcb2cdd0458e341d555fec46e0ae65060c968cf5aa2a48f70b286718"} Jul 07 13:36:49.848541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:49.848498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cmj7v" event={"ID":"041b74ba-8105-469f-8795-1fd332599663","Type":"ContainerStarted","Data":"fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a"} Jul 07 13:36:49.850021 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:49.849994 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" event={"ID":"beaba350-7748-4ae6-8596-181d40e432c9","Type":"ContainerStarted","Data":"33d118525e6c1f5e548b8e9d63fe823fccaad252b99af4298e02c8d0dd7f8a2f"} Jul 07 13:36:49.850143 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:49.850051 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:36:49.851482 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:49.851459 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-pcpfv" event={"ID":"08d5cb0a-5e95-41fe-a073-b0e982df4f43","Type":"ContainerStarted","Data":"712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5"} Jul 07 13:36:49.895931 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:49.895887 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-cmj7v" podStartSLOduration=1.268695942 podStartE2EDuration="5.895874417s" totalImagesPullingTime="4.627178475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:36:49.895293798 +0000 UTC m=+543.896535325" watchObservedRunningTime="2026-07-07 13:36:49.895874417 +0000 UTC m=+543.897115922" Jul 07 13:36:49.953601 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:49.953552 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" podStartSLOduration=1.390157085 podStartE2EDuration="6.95353576s" totalImagesPullingTime="5.563378675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:36:49.951495288 +0000 UTC m=+543.952736814" watchObservedRunningTime="2026-07-07 13:36:49.95353576 +0000 UTC m=+543.954777286" Jul 07 13:36:49.953910 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:49.953891 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-pcpfv" podStartSLOduration=2.24847401 podStartE2EDuration="6.95388531s" totalImagesPullingTime="4.7054113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:36:49.927929769 +0000 UTC m=+543.929171298" watchObservedRunningTime="2026-07-07 13:36:49.95388531 +0000 UTC m=+543.955126836" Jul 07 13:36:49.967528 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:49.967493 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-pcpfv"] Jul 07 13:36:51.860288 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:51.860244 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-pcpfv" podUID="08d5cb0a-5e95-41fe-a073-b0e982df4f43" containerName="authorino" containerID="cri-o://712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5" gracePeriod=30 Jul 07 13:36:52.116943 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.116883 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-pcpfv" Jul 07 13:36:52.187926 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.187900 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf9rf\" (UniqueName: \"kubernetes.io/projected/08d5cb0a-5e95-41fe-a073-b0e982df4f43-kube-api-access-wf9rf\") pod \"08d5cb0a-5e95-41fe-a073-b0e982df4f43\" (UID: \"08d5cb0a-5e95-41fe-a073-b0e982df4f43\") " Jul 07 13:36:52.189889 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.189855 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d5cb0a-5e95-41fe-a073-b0e982df4f43-kube-api-access-wf9rf" (OuterVolumeSpecName: "kube-api-access-wf9rf") pod "08d5cb0a-5e95-41fe-a073-b0e982df4f43" (UID: "08d5cb0a-5e95-41fe-a073-b0e982df4f43"). InnerVolumeSpecName "kube-api-access-wf9rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:36:52.289437 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.289405 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf9rf\" (UniqueName: \"kubernetes.io/projected/08d5cb0a-5e95-41fe-a073-b0e982df4f43-kube-api-access-wf9rf\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:36:52.865794 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.865710 2576 generic.go:356] "Generic (PLEG): container finished" podID="08d5cb0a-5e95-41fe-a073-b0e982df4f43" containerID="712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5" exitCode=0 Jul 07 13:36:52.865794 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.865745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-pcpfv" event={"ID":"08d5cb0a-5e95-41fe-a073-b0e982df4f43","Type":"ContainerDied","Data":"712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5"} Jul 07 13:36:52.865794 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.865755 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-pcpfv" Jul 07 13:36:52.865794 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.865772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-pcpfv" event={"ID":"08d5cb0a-5e95-41fe-a073-b0e982df4f43","Type":"ContainerDied","Data":"ea47e60ddcb2cdd0458e341d555fec46e0ae65060c968cf5aa2a48f70b286718"} Jul 07 13:36:52.865794 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.865793 2576 scope.go:117] "RemoveContainer" containerID="712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5" Jul 07 13:36:52.873855 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.873834 2576 scope.go:117] "RemoveContainer" containerID="712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5" Jul 07 13:36:52.874135 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:36:52.874116 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5\": container with ID starting with 712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5 not found: ID does not exist" containerID="712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5" Jul 07 13:36:52.874198 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.874143 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5"} err="failed to get container status \"712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5\": rpc error: code = NotFound desc = could not find container \"712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5\": container with ID starting with 712d6ca4c22e8a42a61bd268ba45a3ac7312ea2c2d6ccd59cb3e19d892fea5f5 not found: ID does not exist" Jul 07 13:36:52.884630 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.884611 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-pcpfv"] Jul 07 13:36:52.893809 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:52.893789 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-pcpfv"] Jul 07 13:36:54.589125 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:54.589088 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08d5cb0a-5e95-41fe-a073-b0e982df4f43" path="/var/lib/kubelet/pods/08d5cb0a-5e95-41fe-a073-b0e982df4f43/volumes" Jul 07 13:36:59.380245 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:59.380215 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hhwz6"] Jul 07 13:36:59.380638 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:59.380402 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" podUID="beaba350-7748-4ae6-8596-181d40e432c9" containerName="limitador" containerID="cri-o://33d118525e6c1f5e548b8e9d63fe823fccaad252b99af4298e02c8d0dd7f8a2f" gracePeriod=30 Jul 07 13:36:59.382609 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:59.382590 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:36:59.899994 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:59.899958 2576 generic.go:356] "Generic (PLEG): container finished" podID="beaba350-7748-4ae6-8596-181d40e432c9" containerID="33d118525e6c1f5e548b8e9d63fe823fccaad252b99af4298e02c8d0dd7f8a2f" exitCode=0 Jul 07 13:36:59.900138 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:59.900001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" event={"ID":"beaba350-7748-4ae6-8596-181d40e432c9","Type":"ContainerDied","Data":"33d118525e6c1f5e548b8e9d63fe823fccaad252b99af4298e02c8d0dd7f8a2f"} Jul 07 13:36:59.924105 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:36:59.924084 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:37:00.059647 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.059618 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/beaba350-7748-4ae6-8596-181d40e432c9-config-file\") pod \"beaba350-7748-4ae6-8596-181d40e432c9\" (UID: \"beaba350-7748-4ae6-8596-181d40e432c9\") " Jul 07 13:37:00.059827 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.059686 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwdlc\" (UniqueName: \"kubernetes.io/projected/beaba350-7748-4ae6-8596-181d40e432c9-kube-api-access-gwdlc\") pod \"beaba350-7748-4ae6-8596-181d40e432c9\" (UID: \"beaba350-7748-4ae6-8596-181d40e432c9\") " Jul 07 13:37:00.060055 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.060031 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beaba350-7748-4ae6-8596-181d40e432c9-config-file" (OuterVolumeSpecName: "config-file") pod "beaba350-7748-4ae6-8596-181d40e432c9" (UID: "beaba350-7748-4ae6-8596-181d40e432c9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:37:00.061761 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.061744 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beaba350-7748-4ae6-8596-181d40e432c9-kube-api-access-gwdlc" (OuterVolumeSpecName: "kube-api-access-gwdlc") pod "beaba350-7748-4ae6-8596-181d40e432c9" (UID: "beaba350-7748-4ae6-8596-181d40e432c9"). InnerVolumeSpecName "kube-api-access-gwdlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:37:00.160440 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.160402 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/beaba350-7748-4ae6-8596-181d40e432c9-config-file\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:37:00.160440 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.160436 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwdlc\" (UniqueName: \"kubernetes.io/projected/beaba350-7748-4ae6-8596-181d40e432c9-kube-api-access-gwdlc\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:37:00.905487 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.905399 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" Jul 07 13:37:00.905487 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.905407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hhwz6" event={"ID":"beaba350-7748-4ae6-8596-181d40e432c9","Type":"ContainerDied","Data":"1d55d3ab5a01ea2bf540e4afd32d90e8840693f97c61f426cda72c987e7c244f"} Jul 07 13:37:00.905487 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.905472 2576 scope.go:117] "RemoveContainer" containerID="33d118525e6c1f5e548b8e9d63fe823fccaad252b99af4298e02c8d0dd7f8a2f" Jul 07 13:37:00.930409 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.930384 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hhwz6"] Jul 07 13:37:00.937435 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:00.937411 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hhwz6"] Jul 07 13:37:02.590035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:02.590003 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beaba350-7748-4ae6-8596-181d40e432c9" path="/var/lib/kubelet/pods/beaba350-7748-4ae6-8596-181d40e432c9/volumes" Jul 07 13:37:04.671722 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.671688 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-lcwrd"] Jul 07 13:37:04.672217 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.672201 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beaba350-7748-4ae6-8596-181d40e432c9" containerName="limitador" Jul 07 13:37:04.672287 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.672219 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="beaba350-7748-4ae6-8596-181d40e432c9" containerName="limitador" Jul 07 13:37:04.672287 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.672235 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d5cb0a-5e95-41fe-a073-b0e982df4f43" containerName="authorino" Jul 07 13:37:04.672287 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.672244 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d5cb0a-5e95-41fe-a073-b0e982df4f43" containerName="authorino" Jul 07 13:37:04.672373 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.672336 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d5cb0a-5e95-41fe-a073-b0e982df4f43" containerName="authorino" Jul 07 13:37:04.672373 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.672346 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="beaba350-7748-4ae6-8596-181d40e432c9" containerName="limitador" Jul 07 13:37:04.675703 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.675682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-lcwrd" Jul 07 13:37:04.678805 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.678785 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 07 13:37:04.678939 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.678787 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mfxq5\"" Jul 07 13:37:04.682407 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.682381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-lcwrd"] Jul 07 13:37:04.801553 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.801518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/130dfdad-44a1-4dab-8e2b-01c3d2ecf09c-data\") pod \"postgres-69fb4594cf-lcwrd\" (UID: \"130dfdad-44a1-4dab-8e2b-01c3d2ecf09c\") " pod="opendatahub/postgres-69fb4594cf-lcwrd" Jul 07 13:37:04.801721 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.801570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpklw\" (UniqueName: \"kubernetes.io/projected/130dfdad-44a1-4dab-8e2b-01c3d2ecf09c-kube-api-access-cpklw\") pod \"postgres-69fb4594cf-lcwrd\" (UID: \"130dfdad-44a1-4dab-8e2b-01c3d2ecf09c\") " pod="opendatahub/postgres-69fb4594cf-lcwrd" Jul 07 13:37:04.902117 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.902081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpklw\" (UniqueName: \"kubernetes.io/projected/130dfdad-44a1-4dab-8e2b-01c3d2ecf09c-kube-api-access-cpklw\") pod \"postgres-69fb4594cf-lcwrd\" (UID: \"130dfdad-44a1-4dab-8e2b-01c3d2ecf09c\") " pod="opendatahub/postgres-69fb4594cf-lcwrd" Jul 07 13:37:04.902276 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.902251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/130dfdad-44a1-4dab-8e2b-01c3d2ecf09c-data\") pod \"postgres-69fb4594cf-lcwrd\" (UID: \"130dfdad-44a1-4dab-8e2b-01c3d2ecf09c\") " pod="opendatahub/postgres-69fb4594cf-lcwrd" Jul 07 13:37:04.902585 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.902567 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/130dfdad-44a1-4dab-8e2b-01c3d2ecf09c-data\") pod \"postgres-69fb4594cf-lcwrd\" (UID: \"130dfdad-44a1-4dab-8e2b-01c3d2ecf09c\") " pod="opendatahub/postgres-69fb4594cf-lcwrd" Jul 07 13:37:04.913056 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.913028 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpklw\" (UniqueName: \"kubernetes.io/projected/130dfdad-44a1-4dab-8e2b-01c3d2ecf09c-kube-api-access-cpklw\") pod \"postgres-69fb4594cf-lcwrd\" (UID: \"130dfdad-44a1-4dab-8e2b-01c3d2ecf09c\") " pod="opendatahub/postgres-69fb4594cf-lcwrd" Jul 07 13:37:04.987710 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:04.987684 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-lcwrd" Jul 07 13:37:05.121766 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:05.121740 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-lcwrd"] Jul 07 13:37:05.123593 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:37:05.123568 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130dfdad_44a1_4dab_8e2b_01c3d2ecf09c.slice/crio-49d4daa1cbb8d338aa55f36e2bb007c20ec6e7facb6bec8644a184fb3f876610 WatchSource:0}: Error finding container 49d4daa1cbb8d338aa55f36e2bb007c20ec6e7facb6bec8644a184fb3f876610: Status 404 returned error can't find the container with id 49d4daa1cbb8d338aa55f36e2bb007c20ec6e7facb6bec8644a184fb3f876610 Jul 07 13:37:05.933678 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:05.933640 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-lcwrd" event={"ID":"130dfdad-44a1-4dab-8e2b-01c3d2ecf09c","Type":"ContainerStarted","Data":"49d4daa1cbb8d338aa55f36e2bb007c20ec6e7facb6bec8644a184fb3f876610"} Jul 07 13:37:10.967260 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:10.967218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-lcwrd" event={"ID":"130dfdad-44a1-4dab-8e2b-01c3d2ecf09c","Type":"ContainerStarted","Data":"9fbb72355c1242796aec3a449b4bef9167cc244c91c5634e5c4e80d3abab9cb8"} Jul 07 13:37:10.967710 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:10.967312 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-lcwrd" Jul 07 13:37:10.990626 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:10.990578 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-lcwrd" podStartSLOduration=2.187798004 podStartE2EDuration="6.990565254s" totalImagesPullingTime="4.80276725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:37:10.988883688 +0000 UTC m=+564.990125215" watchObservedRunningTime="2026-07-07 13:37:10.990565254 +0000 UTC m=+564.991806781" Jul 07 13:37:17.000963 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:17.000932 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-lcwrd" Jul 07 13:37:17.680819 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:17.680791 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mcdz5"] Jul 07 13:37:17.685902 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:17.685882 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mcdz5" Jul 07 13:37:17.697982 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:17.697961 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mcdz5"] Jul 07 13:37:17.714442 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:17.714418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rsx\" (UniqueName: \"kubernetes.io/projected/d855ec4d-c05e-48fa-92c3-8927afea95f7-kube-api-access-f6rsx\") pod \"authorino-5d4b67c446-mcdz5\" (UID: \"d855ec4d-c05e-48fa-92c3-8927afea95f7\") " pod="kuadrant-system/authorino-5d4b67c446-mcdz5" Jul 07 13:37:17.815696 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:17.815671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rsx\" (UniqueName: \"kubernetes.io/projected/d855ec4d-c05e-48fa-92c3-8927afea95f7-kube-api-access-f6rsx\") pod \"authorino-5d4b67c446-mcdz5\" (UID: \"d855ec4d-c05e-48fa-92c3-8927afea95f7\") " pod="kuadrant-system/authorino-5d4b67c446-mcdz5" Jul 07 13:37:17.825873 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:17.825850 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rsx\" (UniqueName: \"kubernetes.io/projected/d855ec4d-c05e-48fa-92c3-8927afea95f7-kube-api-access-f6rsx\") pod \"authorino-5d4b67c446-mcdz5\" (UID: \"d855ec4d-c05e-48fa-92c3-8927afea95f7\") " pod="kuadrant-system/authorino-5d4b67c446-mcdz5" Jul 07 13:37:17.958564 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:17.958484 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mcdz5"] Jul 07 13:37:17.958735 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:17.958723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mcdz5" Jul 07 13:37:18.111710 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:18.111676 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mcdz5"] Jul 07 13:37:18.114920 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:37:18.114891 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd855ec4d_c05e_48fa_92c3_8927afea95f7.slice/crio-4ef38a03743f46cda20f82d7a6fb00be5a80952b293226d343e47654d72c69b4 WatchSource:0}: Error finding container 4ef38a03743f46cda20f82d7a6fb00be5a80952b293226d343e47654d72c69b4: Status 404 returned error can't find the container with id 4ef38a03743f46cda20f82d7a6fb00be5a80952b293226d343e47654d72c69b4 Jul 07 13:37:19.004100 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:19.004047 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mcdz5" event={"ID":"d855ec4d-c05e-48fa-92c3-8927afea95f7","Type":"ContainerStarted","Data":"095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3"} Jul 07 13:37:19.004100 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:19.004097 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-mcdz5" podUID="d855ec4d-c05e-48fa-92c3-8927afea95f7" containerName="authorino" containerID="cri-o://095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3" gracePeriod=30 Jul 07 13:37:19.004347 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:19.004108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mcdz5" event={"ID":"d855ec4d-c05e-48fa-92c3-8927afea95f7","Type":"ContainerStarted","Data":"4ef38a03743f46cda20f82d7a6fb00be5a80952b293226d343e47654d72c69b4"} Jul 07 13:37:19.022305 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:19.022264 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-mcdz5" podStartSLOduration=1.693407111 podStartE2EDuration="2.022250682s" totalImagesPullingTime="328.843571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:37:19.021361782 +0000 UTC m=+573.022603308" watchObservedRunningTime="2026-07-07 13:37:19.022250682 +0000 UTC m=+573.023492208" Jul 07 13:37:19.283390 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:19.283367 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mcdz5" Jul 07 13:37:19.430996 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:19.430968 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6rsx\" (UniqueName: \"kubernetes.io/projected/d855ec4d-c05e-48fa-92c3-8927afea95f7-kube-api-access-f6rsx\") pod \"d855ec4d-c05e-48fa-92c3-8927afea95f7\" (UID: \"d855ec4d-c05e-48fa-92c3-8927afea95f7\") " Jul 07 13:37:19.432745 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:19.432723 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d855ec4d-c05e-48fa-92c3-8927afea95f7-kube-api-access-f6rsx" (OuterVolumeSpecName: "kube-api-access-f6rsx") pod "d855ec4d-c05e-48fa-92c3-8927afea95f7" (UID: "d855ec4d-c05e-48fa-92c3-8927afea95f7"). InnerVolumeSpecName "kube-api-access-f6rsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:37:19.532295 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:19.532217 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6rsx\" (UniqueName: \"kubernetes.io/projected/d855ec4d-c05e-48fa-92c3-8927afea95f7-kube-api-access-f6rsx\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:37:20.009782 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:20.009748 2576 generic.go:356] "Generic (PLEG): container finished" podID="d855ec4d-c05e-48fa-92c3-8927afea95f7" containerID="095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3" exitCode=0 Jul 07 13:37:20.009921 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:20.009797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mcdz5" event={"ID":"d855ec4d-c05e-48fa-92c3-8927afea95f7","Type":"ContainerDied","Data":"095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3"} Jul 07 13:37:20.009921 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:20.009804 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mcdz5" Jul 07 13:37:20.009921 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:20.009828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mcdz5" event={"ID":"d855ec4d-c05e-48fa-92c3-8927afea95f7","Type":"ContainerDied","Data":"4ef38a03743f46cda20f82d7a6fb00be5a80952b293226d343e47654d72c69b4"} Jul 07 13:37:20.009921 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:20.009844 2576 scope.go:117] "RemoveContainer" containerID="095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3" Jul 07 13:37:20.018190 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:20.018159 2576 scope.go:117] "RemoveContainer" containerID="095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3" Jul 07 13:37:20.018414 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:37:20.018399 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3\": container with ID starting with 095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3 not found: ID does not exist" containerID="095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3" Jul 07 13:37:20.018464 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:20.018421 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3"} err="failed to get container status \"095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3\": rpc error: code = NotFound desc = could not find container \"095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3\": container with ID starting with 095693067d6f107ac1f4bcd1ca34ab44486414ca97079c64d5a4846ccce178a3 not found: ID does not exist" Jul 07 13:37:20.032613 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:20.032588 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mcdz5"] Jul 07 13:37:20.040585 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:20.040566 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mcdz5"] Jul 07 13:37:20.589537 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:20.589508 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d855ec4d-c05e-48fa-92c3-8927afea95f7" path="/var/lib/kubelet/pods/d855ec4d-c05e-48fa-92c3-8927afea95f7/volumes" Jul 07 13:37:22.526401 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:22.526369 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cmj7v"] Jul 07 13:37:22.526828 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:22.526563 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-cmj7v" podUID="041b74ba-8105-469f-8795-1fd332599663" containerName="authorino" containerID="cri-o://fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a" gracePeriod=30 Jul 07 13:37:22.840703 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:22.840680 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cmj7v" Jul 07 13:37:22.859770 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:22.859746 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8wf7\" (UniqueName: \"kubernetes.io/projected/041b74ba-8105-469f-8795-1fd332599663-kube-api-access-h8wf7\") pod \"041b74ba-8105-469f-8795-1fd332599663\" (UID: \"041b74ba-8105-469f-8795-1fd332599663\") " Jul 07 13:37:22.861740 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:22.861717 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041b74ba-8105-469f-8795-1fd332599663-kube-api-access-h8wf7" (OuterVolumeSpecName: "kube-api-access-h8wf7") pod "041b74ba-8105-469f-8795-1fd332599663" (UID: "041b74ba-8105-469f-8795-1fd332599663"). InnerVolumeSpecName "kube-api-access-h8wf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:37:22.960313 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:22.960275 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8wf7\" (UniqueName: \"kubernetes.io/projected/041b74ba-8105-469f-8795-1fd332599663-kube-api-access-h8wf7\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:37:23.024964 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:23.024933 2576 generic.go:356] "Generic (PLEG): container finished" podID="041b74ba-8105-469f-8795-1fd332599663" containerID="fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a" exitCode=0 Jul 07 13:37:23.025116 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:23.024977 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cmj7v" Jul 07 13:37:23.025116 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:23.025016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cmj7v" event={"ID":"041b74ba-8105-469f-8795-1fd332599663","Type":"ContainerDied","Data":"fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a"} Jul 07 13:37:23.025116 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:23.025059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cmj7v" event={"ID":"041b74ba-8105-469f-8795-1fd332599663","Type":"ContainerDied","Data":"097cb7e741dc723fd2e523f4f57430f159e5a23b21d94aae507a7da9c39c5db4"} Jul 07 13:37:23.025116 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:23.025080 2576 scope.go:117] "RemoveContainer" containerID="fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a" Jul 07 13:37:23.033670 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:23.033654 2576 scope.go:117] "RemoveContainer" containerID="fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a" Jul 07 13:37:23.033897 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:37:23.033879 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a\": container with ID starting with fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a not found: ID does not exist" containerID="fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a" Jul 07 13:37:23.033948 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:23.033906 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a"} err="failed to get container status \"fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a\": rpc error: code = NotFound desc = could not find container \"fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a\": container with ID starting with fdbe12b7377367927b7b6feda05ed46beaa7766416ccfd20becd4c8f6e7ab70a not found: ID does not exist" Jul 07 13:37:23.048475 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:23.048422 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cmj7v"] Jul 07 13:37:23.059146 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:23.059127 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cmj7v"] Jul 07 13:37:24.589201 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:24.589151 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041b74ba-8105-469f-8795-1fd332599663" path="/var/lib/kubelet/pods/041b74ba-8105-469f-8795-1fd332599663/volumes" Jul 07 13:37:27.308041 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.308011 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b885c7c54-z54rr"] Jul 07 13:37:27.308509 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.308351 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041b74ba-8105-469f-8795-1fd332599663" containerName="authorino" Jul 07 13:37:27.308509 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.308364 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="041b74ba-8105-469f-8795-1fd332599663" containerName="authorino" Jul 07 13:37:27.308509 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.308389 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d855ec4d-c05e-48fa-92c3-8927afea95f7" containerName="authorino" Jul 07 13:37:27.308509 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.308394 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d855ec4d-c05e-48fa-92c3-8927afea95f7" containerName="authorino" Jul 07 13:37:27.308509 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.308463 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="041b74ba-8105-469f-8795-1fd332599663" containerName="authorino" Jul 07 13:37:27.308509 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.308476 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d855ec4d-c05e-48fa-92c3-8927afea95f7" containerName="authorino" Jul 07 13:37:27.313488 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.313468 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:27.316526 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.316508 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5dhp4\"" Jul 07 13:37:27.316860 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.316840 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 13:37:27.317692 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.317675 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 13:37:27.322426 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.322405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b885c7c54-z54rr"] Jul 07 13:37:27.391064 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.391037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwx2p\" (UniqueName: \"kubernetes.io/projected/29c320b3-0bed-4d0c-948b-d2be969733c2-kube-api-access-rwx2p\") pod \"maas-controller-6b885c7c54-z54rr\" (UID: \"29c320b3-0bed-4d0c-948b-d2be969733c2\") " pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:27.391259 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.391079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29c320b3-0bed-4d0c-948b-d2be969733c2-webhook-cert\") pod \"maas-controller-6b885c7c54-z54rr\" (UID: \"29c320b3-0bed-4d0c-948b-d2be969733c2\") " pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:27.492243 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.492208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwx2p\" (UniqueName: \"kubernetes.io/projected/29c320b3-0bed-4d0c-948b-d2be969733c2-kube-api-access-rwx2p\") pod \"maas-controller-6b885c7c54-z54rr\" (UID: \"29c320b3-0bed-4d0c-948b-d2be969733c2\") " pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:27.492422 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.492301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29c320b3-0bed-4d0c-948b-d2be969733c2-webhook-cert\") pod \"maas-controller-6b885c7c54-z54rr\" (UID: \"29c320b3-0bed-4d0c-948b-d2be969733c2\") " pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:27.492496 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:37:27.492466 2576 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 07 13:37:27.492602 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:37:27.492529 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c320b3-0bed-4d0c-948b-d2be969733c2-webhook-cert podName:29c320b3-0bed-4d0c-948b-d2be969733c2 nodeName:}" failed. No retries permitted until 2026-07-07 13:37:27.992508986 +0000 UTC m=+581.993750491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/29c320b3-0bed-4d0c-948b-d2be969733c2-webhook-cert") pod "maas-controller-6b885c7c54-z54rr" (UID: "29c320b3-0bed-4d0c-948b-d2be969733c2") : secret "maas-controller-webhook-cert" not found Jul 07 13:37:27.494802 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.494778 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d798ccfdb-b7d88"] Jul 07 13:37:27.497666 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.497647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:37:27.510610 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.510592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwx2p\" (UniqueName: \"kubernetes.io/projected/29c320b3-0bed-4d0c-948b-d2be969733c2-kube-api-access-rwx2p\") pod \"maas-controller-6b885c7c54-z54rr\" (UID: \"29c320b3-0bed-4d0c-948b-d2be969733c2\") " pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:27.519783 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.519762 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d798ccfdb-b7d88"] Jul 07 13:37:27.593728 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.593653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/561fe130-aed3-4e03-bc8a-42bd18f5ae83-webhook-cert\") pod \"maas-controller-7d798ccfdb-b7d88\" (UID: \"561fe130-aed3-4e03-bc8a-42bd18f5ae83\") " pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:37:27.593728 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.593687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56jbp\" (UniqueName: \"kubernetes.io/projected/561fe130-aed3-4e03-bc8a-42bd18f5ae83-kube-api-access-56jbp\") pod \"maas-controller-7d798ccfdb-b7d88\" (UID: \"561fe130-aed3-4e03-bc8a-42bd18f5ae83\") " pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:37:27.694964 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.694931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/561fe130-aed3-4e03-bc8a-42bd18f5ae83-webhook-cert\") pod \"maas-controller-7d798ccfdb-b7d88\" (UID: \"561fe130-aed3-4e03-bc8a-42bd18f5ae83\") " pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:37:27.694964 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.694966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56jbp\" (UniqueName: \"kubernetes.io/projected/561fe130-aed3-4e03-bc8a-42bd18f5ae83-kube-api-access-56jbp\") pod \"maas-controller-7d798ccfdb-b7d88\" (UID: \"561fe130-aed3-4e03-bc8a-42bd18f5ae83\") " pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:37:27.697323 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.697297 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/561fe130-aed3-4e03-bc8a-42bd18f5ae83-webhook-cert\") pod \"maas-controller-7d798ccfdb-b7d88\" (UID: \"561fe130-aed3-4e03-bc8a-42bd18f5ae83\") " pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:37:27.705153 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.705126 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56jbp\" (UniqueName: \"kubernetes.io/projected/561fe130-aed3-4e03-bc8a-42bd18f5ae83-kube-api-access-56jbp\") pod \"maas-controller-7d798ccfdb-b7d88\" (UID: \"561fe130-aed3-4e03-bc8a-42bd18f5ae83\") " pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:37:27.808138 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.808096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:37:27.940893 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.940871 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d798ccfdb-b7d88"] Jul 07 13:37:27.943059 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:37:27.943033 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561fe130_aed3_4e03_bc8a_42bd18f5ae83.slice/crio-ad44563db4329fa31dc45ebc1416672ce5f9153c089fd533280f951e24ed6454 WatchSource:0}: Error finding container ad44563db4329fa31dc45ebc1416672ce5f9153c089fd533280f951e24ed6454: Status 404 returned error can't find the container with id ad44563db4329fa31dc45ebc1416672ce5f9153c089fd533280f951e24ed6454 Jul 07 13:37:27.997199 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.997155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29c320b3-0bed-4d0c-948b-d2be969733c2-webhook-cert\") pod \"maas-controller-6b885c7c54-z54rr\" (UID: \"29c320b3-0bed-4d0c-948b-d2be969733c2\") " pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:27.999460 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:27.999438 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29c320b3-0bed-4d0c-948b-d2be969733c2-webhook-cert\") pod \"maas-controller-6b885c7c54-z54rr\" (UID: \"29c320b3-0bed-4d0c-948b-d2be969733c2\") " pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:28.050173 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:28.050137 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" event={"ID":"561fe130-aed3-4e03-bc8a-42bd18f5ae83","Type":"ContainerStarted","Data":"ad44563db4329fa31dc45ebc1416672ce5f9153c089fd533280f951e24ed6454"} Jul 07 13:37:28.223745 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:28.223666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:28.357558 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:28.357528 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b885c7c54-z54rr"] Jul 07 13:37:28.358475 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:37:28.358453 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c320b3_0bed_4d0c_948b_d2be969733c2.slice/crio-624a11435f2d9e12645e3d1bd01ff537f09e925f20c7a37f22af4cec48df87bc WatchSource:0}: Error finding container 624a11435f2d9e12645e3d1bd01ff537f09e925f20c7a37f22af4cec48df87bc: Status 404 returned error can't find the container with id 624a11435f2d9e12645e3d1bd01ff537f09e925f20c7a37f22af4cec48df87bc Jul 07 13:37:29.059239 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:29.059204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b885c7c54-z54rr" event={"ID":"29c320b3-0bed-4d0c-948b-d2be969733c2","Type":"ContainerStarted","Data":"624a11435f2d9e12645e3d1bd01ff537f09e925f20c7a37f22af4cec48df87bc"} Jul 07 13:37:31.071456 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:31.071418 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b885c7c54-z54rr" event={"ID":"29c320b3-0bed-4d0c-948b-d2be969733c2","Type":"ContainerStarted","Data":"c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca"} Jul 07 13:37:31.071905 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:31.071488 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:31.072899 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:31.072878 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" event={"ID":"561fe130-aed3-4e03-bc8a-42bd18f5ae83","Type":"ContainerStarted","Data":"043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed"} Jul 07 13:37:31.072984 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:31.072919 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:37:31.101697 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:31.101648 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b885c7c54-z54rr" podStartSLOduration=2.249127548 podStartE2EDuration="4.101633354s" totalImagesPullingTime="1.852505806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:37:31.099396199 +0000 UTC m=+585.100637726" watchObservedRunningTime="2026-07-07 13:37:31.101633354 +0000 UTC m=+585.102874897" Jul 07 13:37:31.126337 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:31.126285 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" podStartSLOduration=1.867389736 podStartE2EDuration="4.126270238s" totalImagesPullingTime="2.258880502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:37:31.123961662 +0000 UTC m=+585.125203186" watchObservedRunningTime="2026-07-07 13:37:31.126270238 +0000 UTC m=+585.127511766" Jul 07 13:37:42.083012 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.082984 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:37:42.083427 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.083051 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:42.179089 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.179055 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b885c7c54-z54rr"] Jul 07 13:37:42.179364 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.179313 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6b885c7c54-z54rr" podUID="29c320b3-0bed-4d0c-948b-d2be969733c2" containerName="manager" containerID="cri-o://c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca" gracePeriod=30 Jul 07 13:37:42.418834 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.418811 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:42.527747 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.527711 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29c320b3-0bed-4d0c-948b-d2be969733c2-webhook-cert\") pod \"29c320b3-0bed-4d0c-948b-d2be969733c2\" (UID: \"29c320b3-0bed-4d0c-948b-d2be969733c2\") " Jul 07 13:37:42.527940 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.527813 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwx2p\" (UniqueName: \"kubernetes.io/projected/29c320b3-0bed-4d0c-948b-d2be969733c2-kube-api-access-rwx2p\") pod \"29c320b3-0bed-4d0c-948b-d2be969733c2\" (UID: \"29c320b3-0bed-4d0c-948b-d2be969733c2\") " Jul 07 13:37:42.529858 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.529827 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c320b3-0bed-4d0c-948b-d2be969733c2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "29c320b3-0bed-4d0c-948b-d2be969733c2" (UID: "29c320b3-0bed-4d0c-948b-d2be969733c2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 13:37:42.529977 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.529947 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c320b3-0bed-4d0c-948b-d2be969733c2-kube-api-access-rwx2p" (OuterVolumeSpecName: "kube-api-access-rwx2p") pod "29c320b3-0bed-4d0c-948b-d2be969733c2" (UID: "29c320b3-0bed-4d0c-948b-d2be969733c2"). InnerVolumeSpecName "kube-api-access-rwx2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:37:42.629266 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.629239 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwx2p\" (UniqueName: \"kubernetes.io/projected/29c320b3-0bed-4d0c-948b-d2be969733c2-kube-api-access-rwx2p\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:37:42.629266 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:42.629261 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29c320b3-0bed-4d0c-948b-d2be969733c2-webhook-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:37:43.130516 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:43.130484 2576 generic.go:356] "Generic (PLEG): container finished" podID="29c320b3-0bed-4d0c-948b-d2be969733c2" containerID="c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca" exitCode=0 Jul 07 13:37:43.130965 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:43.130527 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b885c7c54-z54rr" Jul 07 13:37:43.130965 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:43.130571 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b885c7c54-z54rr" event={"ID":"29c320b3-0bed-4d0c-948b-d2be969733c2","Type":"ContainerDied","Data":"c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca"} Jul 07 13:37:43.130965 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:43.130634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b885c7c54-z54rr" event={"ID":"29c320b3-0bed-4d0c-948b-d2be969733c2","Type":"ContainerDied","Data":"624a11435f2d9e12645e3d1bd01ff537f09e925f20c7a37f22af4cec48df87bc"} Jul 07 13:37:43.130965 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:43.130670 2576 scope.go:117] "RemoveContainer" containerID="c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca" Jul 07 13:37:43.139053 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:43.139033 2576 scope.go:117] "RemoveContainer" containerID="c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca" Jul 07 13:37:43.139323 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:37:43.139305 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca\": container with ID starting with c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca not found: ID does not exist" containerID="c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca" Jul 07 13:37:43.139382 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:43.139331 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca"} err="failed to get container status \"c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca\": rpc error: code = NotFound desc = could not find container \"c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca\": container with ID starting with c2427f21454d1b7bd050d33e2196ade1e3c461a6fe6c116bf44a86b1e6f94dca not found: ID does not exist" Jul 07 13:37:43.150498 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:43.150472 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b885c7c54-z54rr"] Jul 07 13:37:43.153386 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:43.153367 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b885c7c54-z54rr"] Jul 07 13:37:44.589756 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:37:44.589721 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c320b3-0bed-4d0c-948b-d2be969733c2" path="/var/lib/kubelet/pods/29c320b3-0bed-4d0c-948b-d2be969733c2/volumes" Jul 07 13:38:00.140204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.140161 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-ffb787797-6w54p"] Jul 07 13:38:00.140649 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.140507 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c320b3-0bed-4d0c-948b-d2be969733c2" containerName="manager" Jul 07 13:38:00.140649 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.140519 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c320b3-0bed-4d0c-948b-d2be969733c2" containerName="manager" Jul 07 13:38:00.140649 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.140584 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c320b3-0bed-4d0c-948b-d2be969733c2" containerName="manager" Jul 07 13:38:00.143263 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.143245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.146318 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.146297 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 13:38:00.146426 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.146316 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-v5dbl\"" Jul 07 13:38:00.154599 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.154578 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-ffb787797-6w54p"] Jul 07 13:38:00.264795 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.264770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2xs\" (UniqueName: \"kubernetes.io/projected/87cd98f5-c09f-419b-bc9d-1bf69143d222-kube-api-access-6c2xs\") pod \"maas-api-ffb787797-6w54p\" (UID: \"87cd98f5-c09f-419b-bc9d-1bf69143d222\") " pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.264961 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.264820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cd98f5-c09f-419b-bc9d-1bf69143d222-openshift-service-ca\") pod \"maas-api-ffb787797-6w54p\" (UID: \"87cd98f5-c09f-419b-bc9d-1bf69143d222\") " pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.264961 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.264887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87cd98f5-c09f-419b-bc9d-1bf69143d222-maas-api-tls\") pod \"maas-api-ffb787797-6w54p\" (UID: \"87cd98f5-c09f-419b-bc9d-1bf69143d222\") " pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.366143 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.366113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2xs\" (UniqueName: \"kubernetes.io/projected/87cd98f5-c09f-419b-bc9d-1bf69143d222-kube-api-access-6c2xs\") pod \"maas-api-ffb787797-6w54p\" (UID: \"87cd98f5-c09f-419b-bc9d-1bf69143d222\") " pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.366313 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.366197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cd98f5-c09f-419b-bc9d-1bf69143d222-openshift-service-ca\") pod \"maas-api-ffb787797-6w54p\" (UID: \"87cd98f5-c09f-419b-bc9d-1bf69143d222\") " pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.366313 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.366222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87cd98f5-c09f-419b-bc9d-1bf69143d222-maas-api-tls\") pod \"maas-api-ffb787797-6w54p\" (UID: \"87cd98f5-c09f-419b-bc9d-1bf69143d222\") " pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.366879 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.366848 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cd98f5-c09f-419b-bc9d-1bf69143d222-openshift-service-ca\") pod \"maas-api-ffb787797-6w54p\" (UID: \"87cd98f5-c09f-419b-bc9d-1bf69143d222\") " pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.368702 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.368676 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87cd98f5-c09f-419b-bc9d-1bf69143d222-maas-api-tls\") pod \"maas-api-ffb787797-6w54p\" (UID: \"87cd98f5-c09f-419b-bc9d-1bf69143d222\") " pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.381804 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.381774 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2xs\" (UniqueName: \"kubernetes.io/projected/87cd98f5-c09f-419b-bc9d-1bf69143d222-kube-api-access-6c2xs\") pod \"maas-api-ffb787797-6w54p\" (UID: \"87cd98f5-c09f-419b-bc9d-1bf69143d222\") " pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.453197 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.453118 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:00.593082 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.593059 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-ffb787797-6w54p"] Jul 07 13:38:00.594101 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:38:00.594077 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cd98f5_c09f_419b_bc9d_1bf69143d222.slice/crio-1eee7618b458a10b79050e40b4790cab9fb8a4b5dff50bb9a3fd13bde6950c5c WatchSource:0}: Error finding container 1eee7618b458a10b79050e40b4790cab9fb8a4b5dff50bb9a3fd13bde6950c5c: Status 404 returned error can't find the container with id 1eee7618b458a10b79050e40b4790cab9fb8a4b5dff50bb9a3fd13bde6950c5c Jul 07 13:38:00.665780 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.665751 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-w6qzb"] Jul 07 13:38:00.668832 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.668814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" Jul 07 13:38:00.671685 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.671665 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 07 13:38:00.683531 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.683510 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-w6qzb"] Jul 07 13:38:00.686151 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.686128 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-tn72h"] Jul 07 13:38:00.688532 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.688515 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-tn72h" Jul 07 13:38:00.691496 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.691478 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-57m45\"" Jul 07 13:38:00.704306 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.704257 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-tn72h"] Jul 07 13:38:00.768878 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.768848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k2rm\" (UniqueName: \"kubernetes.io/projected/db1dc435-fd83-456e-82ea-1848a38ac77b-kube-api-access-4k2rm\") pod \"payload-processing-58f86c7798-tn72h\" (UID: \"db1dc435-fd83-456e-82ea-1848a38ac77b\") " pod="openshift-ingress/payload-processing-58f86c7798-tn72h" Jul 07 13:38:00.768994 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.768892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg76r\" (UniqueName: \"kubernetes.io/projected/23b5fcf7-a5c9-4bed-9626-d155169c8e09-kube-api-access-rg76r\") pod \"payload-pre-processing-7964457696-w6qzb\" (UID: \"23b5fcf7-a5c9-4bed-9626-d155169c8e09\") " pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" Jul 07 13:38:00.768994 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.768913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b5fcf7-a5c9-4bed-9626-d155169c8e09-plugins-config-volume\") pod \"payload-pre-processing-7964457696-w6qzb\" (UID: \"23b5fcf7-a5c9-4bed-9626-d155169c8e09\") " pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" Jul 07 13:38:00.768994 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.768949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1dc435-fd83-456e-82ea-1848a38ac77b-plugins-config-volume\") pod \"payload-processing-58f86c7798-tn72h\" (UID: \"db1dc435-fd83-456e-82ea-1848a38ac77b\") " pod="openshift-ingress/payload-processing-58f86c7798-tn72h" Jul 07 13:38:00.870150 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.870117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg76r\" (UniqueName: \"kubernetes.io/projected/23b5fcf7-a5c9-4bed-9626-d155169c8e09-kube-api-access-rg76r\") pod \"payload-pre-processing-7964457696-w6qzb\" (UID: \"23b5fcf7-a5c9-4bed-9626-d155169c8e09\") " pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" Jul 07 13:38:00.870336 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.870180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b5fcf7-a5c9-4bed-9626-d155169c8e09-plugins-config-volume\") pod \"payload-pre-processing-7964457696-w6qzb\" (UID: \"23b5fcf7-a5c9-4bed-9626-d155169c8e09\") " pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" Jul 07 13:38:00.870336 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.870206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1dc435-fd83-456e-82ea-1848a38ac77b-plugins-config-volume\") pod \"payload-processing-58f86c7798-tn72h\" (UID: \"db1dc435-fd83-456e-82ea-1848a38ac77b\") " pod="openshift-ingress/payload-processing-58f86c7798-tn72h" Jul 07 13:38:00.870336 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.870311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k2rm\" (UniqueName: \"kubernetes.io/projected/db1dc435-fd83-456e-82ea-1848a38ac77b-kube-api-access-4k2rm\") pod \"payload-processing-58f86c7798-tn72h\" (UID: \"db1dc435-fd83-456e-82ea-1848a38ac77b\") " pod="openshift-ingress/payload-processing-58f86c7798-tn72h" Jul 07 13:38:00.870863 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.870841 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b5fcf7-a5c9-4bed-9626-d155169c8e09-plugins-config-volume\") pod \"payload-pre-processing-7964457696-w6qzb\" (UID: \"23b5fcf7-a5c9-4bed-9626-d155169c8e09\") " pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" Jul 07 13:38:00.870943 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.870927 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1dc435-fd83-456e-82ea-1848a38ac77b-plugins-config-volume\") pod \"payload-processing-58f86c7798-tn72h\" (UID: \"db1dc435-fd83-456e-82ea-1848a38ac77b\") " pod="openshift-ingress/payload-processing-58f86c7798-tn72h" Jul 07 13:38:00.884861 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.884833 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg76r\" (UniqueName: \"kubernetes.io/projected/23b5fcf7-a5c9-4bed-9626-d155169c8e09-kube-api-access-rg76r\") pod \"payload-pre-processing-7964457696-w6qzb\" (UID: \"23b5fcf7-a5c9-4bed-9626-d155169c8e09\") " pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" Jul 07 13:38:00.899627 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.899602 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k2rm\" (UniqueName: \"kubernetes.io/projected/db1dc435-fd83-456e-82ea-1848a38ac77b-kube-api-access-4k2rm\") pod \"payload-processing-58f86c7798-tn72h\" (UID: \"db1dc435-fd83-456e-82ea-1848a38ac77b\") " pod="openshift-ingress/payload-processing-58f86c7798-tn72h" Jul 07 13:38:00.978854 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:00.978831 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" Jul 07 13:38:01.000744 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:01.000724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-tn72h" Jul 07 13:38:01.126993 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:01.126961 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-w6qzb"] Jul 07 13:38:01.128568 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:38:01.128543 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b5fcf7_a5c9_4bed_9626_d155169c8e09.slice/crio-e636d94c52838a347e4145d3b3edc399cf625eccc64e16d7b11c6ac94c817b55 WatchSource:0}: Error finding container e636d94c52838a347e4145d3b3edc399cf625eccc64e16d7b11c6ac94c817b55: Status 404 returned error can't find the container with id e636d94c52838a347e4145d3b3edc399cf625eccc64e16d7b11c6ac94c817b55 Jul 07 13:38:01.150081 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:01.150059 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-tn72h"] Jul 07 13:38:01.151594 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:38:01.151569 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1dc435_fd83_456e_82ea_1848a38ac77b.slice/crio-fa7e00bfd323b9842efc63bcd0f0c1949a37d34aeadd5ece03be495df34028aa WatchSource:0}: Error finding container fa7e00bfd323b9842efc63bcd0f0c1949a37d34aeadd5ece03be495df34028aa: Status 404 returned error can't find the container with id fa7e00bfd323b9842efc63bcd0f0c1949a37d34aeadd5ece03be495df34028aa Jul 07 13:38:01.211794 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:01.211760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-ffb787797-6w54p" event={"ID":"87cd98f5-c09f-419b-bc9d-1bf69143d222","Type":"ContainerStarted","Data":"1eee7618b458a10b79050e40b4790cab9fb8a4b5dff50bb9a3fd13bde6950c5c"} Jul 07 13:38:01.212789 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:01.212763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-tn72h" event={"ID":"db1dc435-fd83-456e-82ea-1848a38ac77b","Type":"ContainerStarted","Data":"fa7e00bfd323b9842efc63bcd0f0c1949a37d34aeadd5ece03be495df34028aa"} Jul 07 13:38:01.213879 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:01.213858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" event={"ID":"23b5fcf7-a5c9-4bed-9626-d155169c8e09","Type":"ContainerStarted","Data":"e636d94c52838a347e4145d3b3edc399cf625eccc64e16d7b11c6ac94c817b55"} Jul 07 13:38:04.233650 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:04.233614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-ffb787797-6w54p" event={"ID":"87cd98f5-c09f-419b-bc9d-1bf69143d222","Type":"ContainerStarted","Data":"4c10e8f5b43201e6ae0fdd29af04847ef747274fdbb590fbaf1ecc1175f9f134"} Jul 07 13:38:04.234045 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:04.233685 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:04.235134 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:04.235109 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-tn72h" event={"ID":"db1dc435-fd83-456e-82ea-1848a38ac77b","Type":"ContainerStarted","Data":"f14ff6d5a6462d0d17583789c58069805c6c65500e325c47be588d36532c4689"} Jul 07 13:38:04.235222 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:04.235206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-tn72h" Jul 07 13:38:04.236640 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:04.236618 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" event={"ID":"23b5fcf7-a5c9-4bed-9626-d155169c8e09","Type":"ContainerStarted","Data":"7a19715340cd9cf5273ef147b7c329c08ca145af17629237a84ac090cec7c455"} Jul 07 13:38:04.236771 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:04.236754 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" Jul 07 13:38:04.257608 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:04.257545 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-ffb787797-6w54p" podStartSLOduration=1.460463031 podStartE2EDuration="4.257523373s" totalImagesPullingTime="2.797060342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:38:04.255905847 +0000 UTC m=+618.257147375" watchObservedRunningTime="2026-07-07 13:38:04.257523373 +0000 UTC m=+618.258764901" Jul 07 13:38:04.282148 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:04.282072 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" podStartSLOduration=2.022854107 podStartE2EDuration="4.282049606s" totalImagesPullingTime="2.259195499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:38:04.27914296 +0000 UTC m=+618.280384498" watchObservedRunningTime="2026-07-07 13:38:04.282049606 +0000 UTC m=+618.283291137" Jul 07 13:38:04.303252 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:04.303210 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-tn72h" podStartSLOduration=2.067176925 podStartE2EDuration="4.303188904s" totalImagesPullingTime="2.236011979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:38:04.301630805 +0000 UTC m=+618.302872336" watchObservedRunningTime="2026-07-07 13:38:04.303188904 +0000 UTC m=+618.304430423" Jul 07 13:38:09.269717 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:09.269682 2576 generic.go:356] "Generic (PLEG): container finished" podID="dcca19d0-38dd-4c17-b2a6-c9520788d994" containerID="0809274e623e6f8b6ff1608f769fbbd6786bc5eab4c8845ada7902987aa981c1" exitCode=0 Jul 07 13:38:09.270248 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:09.269756 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" event={"ID":"dcca19d0-38dd-4c17-b2a6-c9520788d994","Type":"ContainerDied","Data":"0809274e623e6f8b6ff1608f769fbbd6786bc5eab4c8845ada7902987aa981c1"} Jul 07 13:38:09.270248 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:09.270137 2576 scope.go:117] "RemoveContainer" containerID="0809274e623e6f8b6ff1608f769fbbd6786bc5eab4c8845ada7902987aa981c1" Jul 07 13:38:09.271078 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:09.271051 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:38:09.271140 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:09.271127 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:38:09.271197 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:09.271161 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:38:10.258249 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:10.258215 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-ffb787797-6w54p" Jul 07 13:38:10.279486 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:10.279448 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" event={"ID":"dcca19d0-38dd-4c17-b2a6-c9520788d994","Type":"ContainerStarted","Data":"42fa6db968ff03ccaa93916318aaac4f378baf32a30a3ac6686db2380a36d7f1"} Jul 07 13:38:11.273804 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:11.273759 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:38:11.274437 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:11.274403 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-4qqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 07 13:38:11.274568 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:11.274472 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" podUID="dcca19d0-38dd-4c17-b2a6-c9520788d994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 07 13:38:12.273381 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:12.272900 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-4qqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 07 13:38:12.273381 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:12.272973 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" podUID="dcca19d0-38dd-4c17-b2a6-c9520788d994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 07 13:38:12.798742 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:12.798703 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:38:13.273339 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:13.273301 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-4qqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 07 13:38:13.273640 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:13.273606 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" podUID="dcca19d0-38dd-4c17-b2a6-c9520788d994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 07 13:38:14.273280 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:14.273227 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-4qqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 07 13:38:14.273515 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:14.273290 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" podUID="dcca19d0-38dd-4c17-b2a6-c9520788d994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 07 13:38:15.250331 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:15.250301 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-tn72h" Jul 07 13:38:15.250791 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:15.250372 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-w6qzb" Jul 07 13:38:15.276506 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:15.276481 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:38:15.277372 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:15.277355 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-4qqm6" Jul 07 13:38:30.804779 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.804746 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6"] Jul 07 13:38:30.807424 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.807404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.811925 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.811904 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 13:38:30.812026 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.811945 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 13:38:30.812081 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.812031 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 13:38:30.812081 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.811904 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gvp7x\"" Jul 07 13:38:30.821903 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.821885 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6"] Jul 07 13:38:30.840617 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.840593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.840726 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.840629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.840726 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.840648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.840726 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.840705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.840894 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.840751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc4ds\" (UniqueName: \"kubernetes.io/projected/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-kube-api-access-kc4ds\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.840894 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.840786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.941778 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.941734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.941778 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.941790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.942043 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.941811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.942043 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.941833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.942043 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.941854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc4ds\" (UniqueName: \"kubernetes.io/projected/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-kube-api-access-kc4ds\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.942043 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.941875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.942274 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.942231 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.942274 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.942262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.942483 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.942453 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.944218 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.944192 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.944317 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.944291 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:30.951489 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:30.951465 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc4ds\" (UniqueName: \"kubernetes.io/projected/1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe-kube-api-access-kc4ds\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6\" (UID: \"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:31.118142 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:31.118064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:31.256904 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:31.256825 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6"] Jul 07 13:38:31.258044 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:38:31.258015 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd5d6f6_20f9_48d0_9f1b_951d8e25b0fe.slice/crio-d590ab72d8787ba077d568452c557e77b82872bd346562be6762624eab252cc8 WatchSource:0}: Error finding container d590ab72d8787ba077d568452c557e77b82872bd346562be6762624eab252cc8: Status 404 returned error can't find the container with id d590ab72d8787ba077d568452c557e77b82872bd346562be6762624eab252cc8 Jul 07 13:38:31.259553 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:31.259531 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 13:38:31.375802 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:31.375723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" event={"ID":"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe","Type":"ContainerStarted","Data":"d590ab72d8787ba077d568452c557e77b82872bd346562be6762624eab252cc8"} Jul 07 13:38:33.388048 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:33.388012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" event={"ID":"1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe","Type":"ContainerStarted","Data":"c166a32f3d7040ef797a42f5d81ba3968f92e4463f251e958062b7497dd24830"} Jul 07 13:38:33.408119 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:33.408067 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" podStartSLOduration=2.342564723 podStartE2EDuration="3.40805208s" totalImagesPullingTime="1.065487357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:38:33.406239352 +0000 UTC m=+647.407480873" watchObservedRunningTime="2026-07-07 13:38:33.40805208 +0000 UTC m=+647.409293607" Jul 07 13:38:41.118950 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:41.118910 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:41.119396 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:41.119003 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:41.131467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:41.131441 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:41.440088 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:41.440013 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6" Jul 07 13:38:47.229133 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.229099 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9"] Jul 07 13:38:47.231994 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.231978 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.234991 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.234970 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 07 13:38:47.253706 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.253688 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9"] Jul 07 13:38:47.382234 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.382207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.382389 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.382238 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2ca1d6-24ff-43b1-b612-8ca4930da245-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.382389 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.382268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.382389 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.382337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.382389 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.382372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.382520 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.382428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27lls\" (UniqueName: \"kubernetes.io/projected/6b2ca1d6-24ff-43b1-b612-8ca4930da245-kube-api-access-27lls\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.483852 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.483787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.483852 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.483825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.483852 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.483843 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27lls\" (UniqueName: \"kubernetes.io/projected/6b2ca1d6-24ff-43b1-b612-8ca4930da245-kube-api-access-27lls\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.484071 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.483909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.484071 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.483935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2ca1d6-24ff-43b1-b612-8ca4930da245-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.484071 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.483967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.484255 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.484199 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.484319 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.484262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.484359 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.484325 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.486131 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.486112 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2ca1d6-24ff-43b1-b612-8ca4930da245-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.486458 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.486438 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2ca1d6-24ff-43b1-b612-8ca4930da245-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.493449 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.493429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27lls\" (UniqueName: \"kubernetes.io/projected/6b2ca1d6-24ff-43b1-b612-8ca4930da245-kube-api-access-27lls\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r4vt9\" (UID: \"6b2ca1d6-24ff-43b1-b612-8ca4930da245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.542405 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.542384 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:47.671457 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:47.671431 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9"] Jul 07 13:38:47.673889 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:38:47.673860 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2ca1d6_24ff_43b1_b612_8ca4930da245.slice/crio-e4de41f0af75c4b764679bfbecb7ddd44afc923d35795d4e246cd5a4702eb5b7 WatchSource:0}: Error finding container e4de41f0af75c4b764679bfbecb7ddd44afc923d35795d4e246cd5a4702eb5b7: Status 404 returned error can't find the container with id e4de41f0af75c4b764679bfbecb7ddd44afc923d35795d4e246cd5a4702eb5b7 Jul 07 13:38:48.467003 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:48.466964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" event={"ID":"6b2ca1d6-24ff-43b1-b612-8ca4930da245","Type":"ContainerStarted","Data":"920192e2d5472b527f5c7b56de2311cdee0ac9d83123c576d381bec1a2962635"} Jul 07 13:38:48.467486 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:48.467027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" event={"ID":"6b2ca1d6-24ff-43b1-b612-8ca4930da245","Type":"ContainerStarted","Data":"e4de41f0af75c4b764679bfbecb7ddd44afc923d35795d4e246cd5a4702eb5b7"} Jul 07 13:38:48.488519 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:48.488469 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" podStartSLOduration=1.255552063 podStartE2EDuration="1.48845377s" totalImagesPullingTime="232.901707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:38:48.487073457 +0000 UTC m=+662.488314984" watchObservedRunningTime="2026-07-07 13:38:48.48845377 +0000 UTC m=+662.489695300" Jul 07 13:38:57.542830 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:57.542798 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:57.543225 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:57.542850 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:57.555729 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:57.555708 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:38:58.526379 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:38:58.526351 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r4vt9" Jul 07 13:39:48.727365 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.727264 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx"] Jul 07 13:39:48.730101 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.730086 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.733395 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.733377 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 07 13:39:48.742325 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.742304 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx"] Jul 07 13:39:48.806440 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.806409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.806564 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.806445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhppf\" (UniqueName: \"kubernetes.io/projected/b9159cd8-20b8-492b-9a7a-58652ea68caf-kube-api-access-qhppf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.806564 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.806488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9159cd8-20b8-492b-9a7a-58652ea68caf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.806564 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.806521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.806564 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.806546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.806696 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.806605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.907126 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.907100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.907299 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.907141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.907299 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.907161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhppf\" (UniqueName: \"kubernetes.io/projected/b9159cd8-20b8-492b-9a7a-58652ea68caf-kube-api-access-qhppf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.907299 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.907221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9159cd8-20b8-492b-9a7a-58652ea68caf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.907299 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.907244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.907299 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.907268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.907614 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.907590 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.907784 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.907656 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.908084 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.907822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.910038 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.909810 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9159cd8-20b8-492b-9a7a-58652ea68caf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.910038 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.909935 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9159cd8-20b8-492b-9a7a-58652ea68caf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:48.916467 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:48.916446 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhppf\" (UniqueName: \"kubernetes.io/projected/b9159cd8-20b8-492b-9a7a-58652ea68caf-kube-api-access-qhppf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctxlx\" (UID: \"b9159cd8-20b8-492b-9a7a-58652ea68caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:49.041225 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:49.041123 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:49.182203 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:49.182160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx"] Jul 07 13:39:49.184355 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:39:49.184329 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9159cd8_20b8_492b_9a7a_58652ea68caf.slice/crio-066ef2975075ce4b8fa56db8281144db47cc77613ff2faa814b14cef70b20ae4 WatchSource:0}: Error finding container 066ef2975075ce4b8fa56db8281144db47cc77613ff2faa814b14cef70b20ae4: Status 404 returned error can't find the container with id 066ef2975075ce4b8fa56db8281144db47cc77613ff2faa814b14cef70b20ae4 Jul 07 13:39:49.745709 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:49.745674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" event={"ID":"b9159cd8-20b8-492b-9a7a-58652ea68caf","Type":"ContainerStarted","Data":"3ecea48a836eb9aaa21e86aa5f5fb13a4c85a65015ae489cd2aea36215a8a499"} Jul 07 13:39:49.746058 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:49.745725 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" event={"ID":"b9159cd8-20b8-492b-9a7a-58652ea68caf","Type":"ContainerStarted","Data":"066ef2975075ce4b8fa56db8281144db47cc77613ff2faa814b14cef70b20ae4"} Jul 07 13:39:49.766299 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:49.766251 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" podStartSLOduration=1.590416688 podStartE2EDuration="1.766237112s" totalImagesPullingTime="175.820424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:39:49.765037516 +0000 UTC m=+723.766279042" watchObservedRunningTime="2026-07-07 13:39:49.766237112 +0000 UTC m=+723.767478640" Jul 07 13:39:59.041730 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:59.041694 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:59.042131 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:59.041758 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:59.055556 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:59.055533 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:39:59.803841 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:39:59.803811 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctxlx" Jul 07 13:42:07.785017 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:07.784981 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d798ccfdb-b7d88"] Jul 07 13:42:07.785581 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:07.785250 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" podUID="561fe130-aed3-4e03-bc8a-42bd18f5ae83" containerName="manager" containerID="cri-o://043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed" gracePeriod=30 Jul 07 13:42:08.033671 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.033649 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:42:08.100853 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.100769 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56jbp\" (UniqueName: \"kubernetes.io/projected/561fe130-aed3-4e03-bc8a-42bd18f5ae83-kube-api-access-56jbp\") pod \"561fe130-aed3-4e03-bc8a-42bd18f5ae83\" (UID: \"561fe130-aed3-4e03-bc8a-42bd18f5ae83\") " Jul 07 13:42:08.100853 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.100842 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/561fe130-aed3-4e03-bc8a-42bd18f5ae83-webhook-cert\") pod \"561fe130-aed3-4e03-bc8a-42bd18f5ae83\" (UID: \"561fe130-aed3-4e03-bc8a-42bd18f5ae83\") " Jul 07 13:42:08.102807 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.102773 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561fe130-aed3-4e03-bc8a-42bd18f5ae83-kube-api-access-56jbp" (OuterVolumeSpecName: "kube-api-access-56jbp") pod "561fe130-aed3-4e03-bc8a-42bd18f5ae83" (UID: "561fe130-aed3-4e03-bc8a-42bd18f5ae83"). InnerVolumeSpecName "kube-api-access-56jbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:42:08.102917 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.102875 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561fe130-aed3-4e03-bc8a-42bd18f5ae83-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "561fe130-aed3-4e03-bc8a-42bd18f5ae83" (UID: "561fe130-aed3-4e03-bc8a-42bd18f5ae83"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 13:42:08.201668 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.201636 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56jbp\" (UniqueName: \"kubernetes.io/projected/561fe130-aed3-4e03-bc8a-42bd18f5ae83-kube-api-access-56jbp\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:42:08.201668 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.201665 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/561fe130-aed3-4e03-bc8a-42bd18f5ae83-webhook-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:42:08.412120 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.412034 2576 generic.go:356] "Generic (PLEG): container finished" podID="561fe130-aed3-4e03-bc8a-42bd18f5ae83" containerID="043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed" exitCode=0 Jul 07 13:42:08.412294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.412119 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" event={"ID":"561fe130-aed3-4e03-bc8a-42bd18f5ae83","Type":"ContainerDied","Data":"043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed"} Jul 07 13:42:08.412294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.412139 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" Jul 07 13:42:08.412294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.412176 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d798ccfdb-b7d88" event={"ID":"561fe130-aed3-4e03-bc8a-42bd18f5ae83","Type":"ContainerDied","Data":"ad44563db4329fa31dc45ebc1416672ce5f9153c089fd533280f951e24ed6454"} Jul 07 13:42:08.412294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.412186 2576 scope.go:117] "RemoveContainer" containerID="043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed" Jul 07 13:42:08.421861 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.421842 2576 scope.go:117] "RemoveContainer" containerID="043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed" Jul 07 13:42:08.422121 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:42:08.422097 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed\": container with ID starting with 043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed not found: ID does not exist" containerID="043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed" Jul 07 13:42:08.422241 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.422124 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed"} err="failed to get container status \"043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed\": rpc error: code = NotFound desc = could not find container \"043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed\": container with ID starting with 043ec24bb5f7c07d9f80356ebb12e4a88130a7b0d88d1c31c5e16b5cd51416ed not found: ID does not exist" Jul 07 13:42:08.437448 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.437428 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d798ccfdb-b7d88"] Jul 07 13:42:08.442957 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.442934 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d798ccfdb-b7d88"] Jul 07 13:42:08.453577 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:42:08.453554 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561fe130_aed3_4e03_bc8a_42bd18f5ae83.slice/crio-ad44563db4329fa31dc45ebc1416672ce5f9153c089fd533280f951e24ed6454\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561fe130_aed3_4e03_bc8a_42bd18f5ae83.slice\": RecentStats: unable to find data in memory cache]" Jul 07 13:42:08.590365 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:08.590336 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="561fe130-aed3-4e03-bc8a-42bd18f5ae83" path="/var/lib/kubelet/pods/561fe130-aed3-4e03-bc8a-42bd18f5ae83/volumes" Jul 07 13:42:09.280851 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.280820 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d798ccfdb-x8k6c"] Jul 07 13:42:09.281234 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.281220 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="561fe130-aed3-4e03-bc8a-42bd18f5ae83" containerName="manager" Jul 07 13:42:09.281282 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.281236 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="561fe130-aed3-4e03-bc8a-42bd18f5ae83" containerName="manager" Jul 07 13:42:09.281317 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.281296 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="561fe130-aed3-4e03-bc8a-42bd18f5ae83" containerName="manager" Jul 07 13:42:09.284437 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.284417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" Jul 07 13:42:09.287507 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.287478 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 13:42:09.287614 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.287538 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 13:42:09.289207 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.289192 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5dhp4\"" Jul 07 13:42:09.295717 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.295696 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d798ccfdb-x8k6c"] Jul 07 13:42:09.411584 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.411555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlsw\" (UniqueName: \"kubernetes.io/projected/5c4dab78-b0c0-4626-9a73-a0fa4ab64b40-kube-api-access-qxlsw\") pod \"maas-controller-7d798ccfdb-x8k6c\" (UID: \"5c4dab78-b0c0-4626-9a73-a0fa4ab64b40\") " pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" Jul 07 13:42:09.411725 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.411611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c4dab78-b0c0-4626-9a73-a0fa4ab64b40-webhook-cert\") pod \"maas-controller-7d798ccfdb-x8k6c\" (UID: \"5c4dab78-b0c0-4626-9a73-a0fa4ab64b40\") " pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" Jul 07 13:42:09.512742 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.512710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlsw\" (UniqueName: \"kubernetes.io/projected/5c4dab78-b0c0-4626-9a73-a0fa4ab64b40-kube-api-access-qxlsw\") pod \"maas-controller-7d798ccfdb-x8k6c\" (UID: \"5c4dab78-b0c0-4626-9a73-a0fa4ab64b40\") " pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" Jul 07 13:42:09.512925 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.512793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c4dab78-b0c0-4626-9a73-a0fa4ab64b40-webhook-cert\") pod \"maas-controller-7d798ccfdb-x8k6c\" (UID: \"5c4dab78-b0c0-4626-9a73-a0fa4ab64b40\") " pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" Jul 07 13:42:09.515176 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.515147 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c4dab78-b0c0-4626-9a73-a0fa4ab64b40-webhook-cert\") pod \"maas-controller-7d798ccfdb-x8k6c\" (UID: \"5c4dab78-b0c0-4626-9a73-a0fa4ab64b40\") " pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" Jul 07 13:42:09.522542 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.522519 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlsw\" (UniqueName: \"kubernetes.io/projected/5c4dab78-b0c0-4626-9a73-a0fa4ab64b40-kube-api-access-qxlsw\") pod \"maas-controller-7d798ccfdb-x8k6c\" (UID: \"5c4dab78-b0c0-4626-9a73-a0fa4ab64b40\") " pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" Jul 07 13:42:09.594476 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.594416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" Jul 07 13:42:09.723321 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:09.723299 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d798ccfdb-x8k6c"] Jul 07 13:42:09.725233 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:42:09.725206 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4dab78_b0c0_4626_9a73_a0fa4ab64b40.slice/crio-c3c8e63a19f82f3295164ba3359b953a8089e75beb4764172318ed2a1f523f21 WatchSource:0}: Error finding container c3c8e63a19f82f3295164ba3359b953a8089e75beb4764172318ed2a1f523f21: Status 404 returned error can't find the container with id c3c8e63a19f82f3295164ba3359b953a8089e75beb4764172318ed2a1f523f21 Jul 07 13:42:10.423561 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:10.423523 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" event={"ID":"5c4dab78-b0c0-4626-9a73-a0fa4ab64b40","Type":"ContainerStarted","Data":"b8b2606adac65ffc435ca21761b9b595cf7ae7e17afe3e58c3b3d688c9212801"} Jul 07 13:42:10.423996 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:10.423580 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" event={"ID":"5c4dab78-b0c0-4626-9a73-a0fa4ab64b40","Type":"ContainerStarted","Data":"c3c8e63a19f82f3295164ba3359b953a8089e75beb4764172318ed2a1f523f21"} Jul 07 13:42:10.423996 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:10.423601 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" Jul 07 13:42:10.446553 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:10.446507 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" podStartSLOduration=0.934166244 podStartE2EDuration="1.446491988s" totalImagesPullingTime="512.325744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:42:10.4436082 +0000 UTC m=+864.444849726" watchObservedRunningTime="2026-07-07 13:42:10.446491988 +0000 UTC m=+864.447733515" Jul 07 13:42:21.434459 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:42:21.434429 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d798ccfdb-x8k6c" Jul 07 13:45:00.168319 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:00.168283 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29723865-zr2j5"] Jul 07 13:45:00.171301 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:00.171285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723865-zr2j5" Jul 07 13:45:00.183857 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:00.183835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z9qt\" (UniqueName: \"kubernetes.io/projected/ca5f68e4-9118-4fa2-b23a-466d07e29f74-kube-api-access-5z9qt\") pod \"maas-api-key-cleanup-29723865-zr2j5\" (UID: \"ca5f68e4-9118-4fa2-b23a-466d07e29f74\") " pod="opendatahub/maas-api-key-cleanup-29723865-zr2j5" Jul 07 13:45:00.193001 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:00.192977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723865-zr2j5"] Jul 07 13:45:00.284735 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:00.284704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z9qt\" (UniqueName: \"kubernetes.io/projected/ca5f68e4-9118-4fa2-b23a-466d07e29f74-kube-api-access-5z9qt\") pod \"maas-api-key-cleanup-29723865-zr2j5\" (UID: \"ca5f68e4-9118-4fa2-b23a-466d07e29f74\") " pod="opendatahub/maas-api-key-cleanup-29723865-zr2j5" Jul 07 13:45:00.295718 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:00.295684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z9qt\" (UniqueName: \"kubernetes.io/projected/ca5f68e4-9118-4fa2-b23a-466d07e29f74-kube-api-access-5z9qt\") pod \"maas-api-key-cleanup-29723865-zr2j5\" (UID: \"ca5f68e4-9118-4fa2-b23a-466d07e29f74\") " pod="opendatahub/maas-api-key-cleanup-29723865-zr2j5" Jul 07 13:45:00.481020 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:00.480994 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723865-zr2j5" Jul 07 13:45:00.619811 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:00.619783 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723865-zr2j5"] Jul 07 13:45:00.623502 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:45:00.623474 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5f68e4_9118_4fa2_b23a_466d07e29f74.slice/crio-e0e6eacc77323d00b60bc8bf195fd787674984b01bb08299acd23f732eb08492 WatchSource:0}: Error finding container e0e6eacc77323d00b60bc8bf195fd787674984b01bb08299acd23f732eb08492: Status 404 returned error can't find the container with id e0e6eacc77323d00b60bc8bf195fd787674984b01bb08299acd23f732eb08492 Jul 07 13:45:00.625309 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:00.625294 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 13:45:01.201878 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:01.201840 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723865-zr2j5" event={"ID":"ca5f68e4-9118-4fa2-b23a-466d07e29f74","Type":"ContainerStarted","Data":"e0e6eacc77323d00b60bc8bf195fd787674984b01bb08299acd23f732eb08492"} Jul 07 13:45:04.217949 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:04.217862 2576 generic.go:356] "Generic (PLEG): container finished" podID="ca5f68e4-9118-4fa2-b23a-466d07e29f74" containerID="005a81a3c8aa09060136b267d0044b2a11034cc1aa2a576f4000f99c15cde2c6" exitCode=0 Jul 07 13:45:04.218334 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:04.217943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723865-zr2j5" event={"ID":"ca5f68e4-9118-4fa2-b23a-466d07e29f74","Type":"ContainerDied","Data":"005a81a3c8aa09060136b267d0044b2a11034cc1aa2a576f4000f99c15cde2c6"} Jul 07 13:45:05.364827 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:05.364803 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723865-zr2j5" Jul 07 13:45:05.429755 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:05.429720 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z9qt\" (UniqueName: \"kubernetes.io/projected/ca5f68e4-9118-4fa2-b23a-466d07e29f74-kube-api-access-5z9qt\") pod \"ca5f68e4-9118-4fa2-b23a-466d07e29f74\" (UID: \"ca5f68e4-9118-4fa2-b23a-466d07e29f74\") " Jul 07 13:45:05.431753 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:05.431726 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5f68e4-9118-4fa2-b23a-466d07e29f74-kube-api-access-5z9qt" (OuterVolumeSpecName: "kube-api-access-5z9qt") pod "ca5f68e4-9118-4fa2-b23a-466d07e29f74" (UID: "ca5f68e4-9118-4fa2-b23a-466d07e29f74"). InnerVolumeSpecName "kube-api-access-5z9qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:45:05.530601 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:05.530563 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z9qt\" (UniqueName: \"kubernetes.io/projected/ca5f68e4-9118-4fa2-b23a-466d07e29f74-kube-api-access-5z9qt\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:45:06.228596 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:06.228569 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723865-zr2j5" Jul 07 13:45:06.228759 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:06.228601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723865-zr2j5" event={"ID":"ca5f68e4-9118-4fa2-b23a-466d07e29f74","Type":"ContainerDied","Data":"e0e6eacc77323d00b60bc8bf195fd787674984b01bb08299acd23f732eb08492"} Jul 07 13:45:06.228759 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:45:06.228645 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e6eacc77323d00b60bc8bf195fd787674984b01bb08299acd23f732eb08492" Jul 07 13:53:58.445778 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.445751 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg"] Jul 07 13:53:58.446348 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.446330 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5f68e4-9118-4fa2-b23a-466d07e29f74" containerName="cleanup" Jul 07 13:53:58.446400 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.446350 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5f68e4-9118-4fa2-b23a-466d07e29f74" containerName="cleanup" Jul 07 13:53:58.446448 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.446439 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5f68e4-9118-4fa2-b23a-466d07e29f74" containerName="cleanup" Jul 07 13:53:58.449387 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.449373 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.453118 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.453097 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-987cf14b-openshift-default-dockercfg-sjm96\"" Jul 07 13:53:58.483046 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.483016 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg"] Jul 07 13:53:58.553568 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.553533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-socket\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.553720 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.553575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-data\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.553720 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.553654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk6sw\" (UniqueName: \"kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-kube-api-access-bk6sw\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.553720 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.553694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-podinfo\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.553720 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.553718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-envoy\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.554035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.553744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-token\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.554035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.553793 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-credential-socket\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.554035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.553904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-certs\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.554035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.553994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c123df96-03ea-49e1-b7c7-262879ac34bb-istiod-ca-cert\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655079 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-podinfo\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655240 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-envoy\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655240 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-token\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655240 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-credential-socket\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655379 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-certs\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655435 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c123df96-03ea-49e1-b7c7-262879ac34bb-istiod-ca-cert\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655495 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-socket\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655552 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-data\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655604 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655547 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-credential-socket\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655604 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk6sw\" (UniqueName: \"kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-kube-api-access-bk6sw\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655773 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-certs\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.655862 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.655806 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-data\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.656037 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.656006 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-socket\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.656309 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.656284 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c123df96-03ea-49e1-b7c7-262879ac34bb-istiod-ca-cert\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.657341 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.657318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-envoy\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.657680 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.657655 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-podinfo\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.667517 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.667496 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-token\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.668970 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.668943 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk6sw\" (UniqueName: \"kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-kube-api-access-bk6sw\") pod \"e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.759848 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.759829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:53:58.901466 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.901440 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg"] Jul 07 13:53:58.903147 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:53:58.903121 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc123df96_03ea_49e1_b7c7_262879ac34bb.slice/crio-46f387a382a10ec475e286ff3d44664cb4d4daf7cf37290fcf73d3ee9886ed8c WatchSource:0}: Error finding container 46f387a382a10ec475e286ff3d44664cb4d4daf7cf37290fcf73d3ee9886ed8c: Status 404 returned error can't find the container with id 46f387a382a10ec475e286ff3d44664cb4d4daf7cf37290fcf73d3ee9886ed8c Jul 07 13:53:58.905184 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.905137 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:53:58.905277 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.905216 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:53:58.905277 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:58.905270 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:53:59.711891 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.711855 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv"] Jul 07 13:53:59.715364 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.715345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:53:59.718541 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.718525 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-987cf14b-serving-cert\"" Jul 07 13:53:59.732944 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.732924 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv"] Jul 07 13:53:59.765500 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.765475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af895fe-d12d-4ccb-a81b-695c809eac90-openshift-service-ca\") pod \"maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:53:59.765630 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.765521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzk64\" (UniqueName: \"kubernetes.io/projected/5af895fe-d12d-4ccb-a81b-695c809eac90-kube-api-access-lzk64\") pod \"maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:53:59.765630 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.765616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af895fe-d12d-4ccb-a81b-695c809eac90-maas-api-tls\") pod \"maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:53:59.765929 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.765910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" event={"ID":"c123df96-03ea-49e1-b7c7-262879ac34bb","Type":"ContainerStarted","Data":"5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c"} Jul 07 13:53:59.765971 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.765948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" event={"ID":"c123df96-03ea-49e1-b7c7-262879ac34bb","Type":"ContainerStarted","Data":"46f387a382a10ec475e286ff3d44664cb4d4daf7cf37290fcf73d3ee9886ed8c"} Jul 07 13:53:59.795068 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.795026 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" podStartSLOduration=1.795013127 podStartE2EDuration="1.795013127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:53:59.792662776 +0000 UTC m=+1573.793904327" watchObservedRunningTime="2026-07-07 13:53:59.795013127 +0000 UTC m=+1573.796254653" Jul 07 13:53:59.866814 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.866786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzk64\" (UniqueName: \"kubernetes.io/projected/5af895fe-d12d-4ccb-a81b-695c809eac90-kube-api-access-lzk64\") pod \"maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:53:59.867132 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.867111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af895fe-d12d-4ccb-a81b-695c809eac90-maas-api-tls\") pod \"maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:53:59.867249 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.867214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af895fe-d12d-4ccb-a81b-695c809eac90-openshift-service-ca\") pod \"maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:53:59.867310 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:53:59.867286 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-987cf14b-serving-cert: secret "maas-api-e2e-ait-987cf14b-serving-cert" not found Jul 07 13:53:59.867358 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:53:59.867344 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5af895fe-d12d-4ccb-a81b-695c809eac90-maas-api-tls podName:5af895fe-d12d-4ccb-a81b-695c809eac90 nodeName:}" failed. No retries permitted until 2026-07-07 13:54:00.367328258 +0000 UTC m=+1574.368569767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5af895fe-d12d-4ccb-a81b-695c809eac90-maas-api-tls") pod "maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" (UID: "5af895fe-d12d-4ccb-a81b-695c809eac90") : secret "maas-api-e2e-ait-987cf14b-serving-cert" not found Jul 07 13:53:59.867820 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.867804 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af895fe-d12d-4ccb-a81b-695c809eac90-openshift-service-ca\") pod \"maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:53:59.877563 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:53:59.877527 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzk64\" (UniqueName: \"kubernetes.io/projected/5af895fe-d12d-4ccb-a81b-695c809eac90-kube-api-access-lzk64\") pod \"maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:54:00.371834 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:00.371799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af895fe-d12d-4ccb-a81b-695c809eac90-maas-api-tls\") pod \"maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:54:00.374182 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:00.374144 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af895fe-d12d-4ccb-a81b-695c809eac90-maas-api-tls\") pod \"maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:54:00.625648 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:00.625572 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:54:00.761029 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:00.760996 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:54:00.761882 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:00.761858 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv"] Jul 07 13:54:00.762386 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:54:00.762348 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af895fe_d12d_4ccb_a81b_695c809eac90.slice/crio-4d46db196a55fb74ded12d1050dea1dd6b0281a2914740a9fa0e983e26462404 WatchSource:0}: Error finding container 4d46db196a55fb74ded12d1050dea1dd6b0281a2914740a9fa0e983e26462404: Status 404 returned error can't find the container with id 4d46db196a55fb74ded12d1050dea1dd6b0281a2914740a9fa0e983e26462404 Jul 07 13:54:00.764149 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:00.764128 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 13:54:00.766194 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:00.766159 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:54:00.771948 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:00.771924 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" event={"ID":"5af895fe-d12d-4ccb-a81b-695c809eac90","Type":"ContainerStarted","Data":"4d46db196a55fb74ded12d1050dea1dd6b0281a2914740a9fa0e983e26462404"} Jul 07 13:54:00.772079 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:00.772059 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:54:00.772810 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:00.772795 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:54:01.777773 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:01.777735 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" event={"ID":"5af895fe-d12d-4ccb-a81b-695c809eac90","Type":"ContainerStarted","Data":"651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb"} Jul 07 13:54:01.778154 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:01.777925 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:54:01.805767 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:01.805691 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" podStartSLOduration=2.183741129 podStartE2EDuration="2.805664578s" totalImagesPullingTime="621.923449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:54:01.803559189 +0000 UTC m=+1575.804800716" watchObservedRunningTime="2026-07-07 13:54:01.805664578 +0000 UTC m=+1575.806906106" Jul 07 13:54:04.970151 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:04.970113 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg"] Jul 07 13:54:04.971017 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:04.970961 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" podUID="c123df96-03ea-49e1-b7c7-262879ac34bb" containerName="istio-proxy" containerID="cri-o://5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c" gracePeriod=30 Jul 07 13:54:05.340334 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.340303 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv"] Jul 07 13:54:05.340550 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.340472 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" podUID="5af895fe-d12d-4ccb-a81b-695c809eac90" containerName="maas-api" containerID="cri-o://651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb" gracePeriod=30 Jul 07 13:54:05.632063 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.632041 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:54:05.723239 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.723208 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af895fe-d12d-4ccb-a81b-695c809eac90-openshift-service-ca\") pod \"5af895fe-d12d-4ccb-a81b-695c809eac90\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " Jul 07 13:54:05.723401 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.723267 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzk64\" (UniqueName: \"kubernetes.io/projected/5af895fe-d12d-4ccb-a81b-695c809eac90-kube-api-access-lzk64\") pod \"5af895fe-d12d-4ccb-a81b-695c809eac90\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " Jul 07 13:54:05.723401 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.723307 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af895fe-d12d-4ccb-a81b-695c809eac90-maas-api-tls\") pod \"5af895fe-d12d-4ccb-a81b-695c809eac90\" (UID: \"5af895fe-d12d-4ccb-a81b-695c809eac90\") " Jul 07 13:54:05.723617 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.723592 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af895fe-d12d-4ccb-a81b-695c809eac90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5af895fe-d12d-4ccb-a81b-695c809eac90" (UID: "5af895fe-d12d-4ccb-a81b-695c809eac90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:54:05.725381 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.725361 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af895fe-d12d-4ccb-a81b-695c809eac90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5af895fe-d12d-4ccb-a81b-695c809eac90" (UID: "5af895fe-d12d-4ccb-a81b-695c809eac90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 13:54:05.725381 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.725368 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af895fe-d12d-4ccb-a81b-695c809eac90-kube-api-access-lzk64" (OuterVolumeSpecName: "kube-api-access-lzk64") pod "5af895fe-d12d-4ccb-a81b-695c809eac90" (UID: "5af895fe-d12d-4ccb-a81b-695c809eac90"). InnerVolumeSpecName "kube-api-access-lzk64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:54:05.800316 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.800284 2576 generic.go:356] "Generic (PLEG): container finished" podID="5af895fe-d12d-4ccb-a81b-695c809eac90" containerID="651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb" exitCode=0 Jul 07 13:54:05.800447 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.800319 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" event={"ID":"5af895fe-d12d-4ccb-a81b-695c809eac90","Type":"ContainerDied","Data":"651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb"} Jul 07 13:54:05.800447 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.800324 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" Jul 07 13:54:05.800447 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.800367 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv" event={"ID":"5af895fe-d12d-4ccb-a81b-695c809eac90","Type":"ContainerDied","Data":"4d46db196a55fb74ded12d1050dea1dd6b0281a2914740a9fa0e983e26462404"} Jul 07 13:54:05.800447 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.800390 2576 scope.go:117] "RemoveContainer" containerID="651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb" Jul 07 13:54:05.809800 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.809785 2576 scope.go:117] "RemoveContainer" containerID="651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb" Jul 07 13:54:05.810013 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:54:05.809996 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb\": container with ID starting with 651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb not found: ID does not exist" containerID="651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb" Jul 07 13:54:05.810058 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.810021 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb"} err="failed to get container status \"651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb\": rpc error: code = NotFound desc = could not find container \"651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb\": container with ID starting with 651a20ac1a3abbb4b5efef25cb03b3b065ef81eea95b5dede519ec14243fd9eb not found: ID does not exist" Jul 07 13:54:05.824485 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.824462 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af895fe-d12d-4ccb-a81b-695c809eac90-openshift-service-ca\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:05.824485 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.824481 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzk64\" (UniqueName: \"kubernetes.io/projected/5af895fe-d12d-4ccb-a81b-695c809eac90-kube-api-access-lzk64\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:05.824597 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.824491 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af895fe-d12d-4ccb-a81b-695c809eac90-maas-api-tls\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:05.829093 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.829070 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv"] Jul 07 13:54:05.837702 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:05.837682 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-987cf14b-dccb4bfc6-4n7rv"] Jul 07 13:54:06.590036 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:06.590006 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af895fe-d12d-4ccb-a81b-695c809eac90" path="/var/lib/kubelet/pods/5af895fe-d12d-4ccb-a81b-695c809eac90/volumes" Jul 07 13:54:11.028641 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.028616 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:54:11.071045 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.070979 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-credential-socket\") pod \"c123df96-03ea-49e1-b7c7-262879ac34bb\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " Jul 07 13:54:11.071045 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.071043 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-token\") pod \"c123df96-03ea-49e1-b7c7-262879ac34bb\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " Jul 07 13:54:11.071224 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.071083 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-certs\") pod \"c123df96-03ea-49e1-b7c7-262879ac34bb\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " Jul 07 13:54:11.071294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.071266 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c123df96-03ea-49e1-b7c7-262879ac34bb" (UID: "c123df96-03ea-49e1-b7c7-262879ac34bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:11.071459 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.071433 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c123df96-03ea-49e1-b7c7-262879ac34bb" (UID: "c123df96-03ea-49e1-b7c7-262879ac34bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:11.073054 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.073033 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "c123df96-03ea-49e1-b7c7-262879ac34bb" (UID: "c123df96-03ea-49e1-b7c7-262879ac34bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:54:11.172392 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.172360 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-data\") pod \"c123df96-03ea-49e1-b7c7-262879ac34bb\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " Jul 07 13:54:11.172540 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.172422 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk6sw\" (UniqueName: \"kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-kube-api-access-bk6sw\") pod \"c123df96-03ea-49e1-b7c7-262879ac34bb\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " Jul 07 13:54:11.172540 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.172453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c123df96-03ea-49e1-b7c7-262879ac34bb-istiod-ca-cert\") pod \"c123df96-03ea-49e1-b7c7-262879ac34bb\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " Jul 07 13:54:11.172540 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.172503 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-socket\") pod \"c123df96-03ea-49e1-b7c7-262879ac34bb\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " Jul 07 13:54:11.172540 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.172523 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-envoy\") pod \"c123df96-03ea-49e1-b7c7-262879ac34bb\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " Jul 07 13:54:11.172757 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.172650 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-podinfo\") pod \"c123df96-03ea-49e1-b7c7-262879ac34bb\" (UID: \"c123df96-03ea-49e1-b7c7-262879ac34bb\") " Jul 07 13:54:11.172830 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.172800 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c123df96-03ea-49e1-b7c7-262879ac34bb" (UID: "c123df96-03ea-49e1-b7c7-262879ac34bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:11.172830 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.172817 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c123df96-03ea-49e1-b7c7-262879ac34bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c123df96-03ea-49e1-b7c7-262879ac34bb" (UID: "c123df96-03ea-49e1-b7c7-262879ac34bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:54:11.173128 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.173110 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c123df96-03ea-49e1-b7c7-262879ac34bb-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:11.173233 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.173131 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:11.173233 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.173145 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:11.173233 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.173158 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:11.173233 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.173183 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:11.173233 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.173207 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "c123df96-03ea-49e1-b7c7-262879ac34bb" (UID: "c123df96-03ea-49e1-b7c7-262879ac34bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:11.174656 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.174633 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c123df96-03ea-49e1-b7c7-262879ac34bb" (UID: "c123df96-03ea-49e1-b7c7-262879ac34bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:11.174740 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.174682 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-kube-api-access-bk6sw" (OuterVolumeSpecName: "kube-api-access-bk6sw") pod "c123df96-03ea-49e1-b7c7-262879ac34bb" (UID: "c123df96-03ea-49e1-b7c7-262879ac34bb"). InnerVolumeSpecName "kube-api-access-bk6sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:54:11.174740 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.174712 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c123df96-03ea-49e1-b7c7-262879ac34bb" (UID: "c123df96-03ea-49e1-b7c7-262879ac34bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 13:54:11.273926 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.273904 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:11.273926 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.273923 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:11.274055 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.273933 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c123df96-03ea-49e1-b7c7-262879ac34bb-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:11.274055 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.273942 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk6sw\" (UniqueName: \"kubernetes.io/projected/c123df96-03ea-49e1-b7c7-262879ac34bb-kube-api-access-bk6sw\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:11.831291 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.831255 2576 generic.go:356] "Generic (PLEG): container finished" podID="c123df96-03ea-49e1-b7c7-262879ac34bb" containerID="5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c" exitCode=0 Jul 07 13:54:11.831479 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.831303 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" Jul 07 13:54:11.831479 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.831302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" event={"ID":"c123df96-03ea-49e1-b7c7-262879ac34bb","Type":"ContainerDied","Data":"5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c"} Jul 07 13:54:11.831479 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.831437 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg" event={"ID":"c123df96-03ea-49e1-b7c7-262879ac34bb","Type":"ContainerDied","Data":"46f387a382a10ec475e286ff3d44664cb4d4daf7cf37290fcf73d3ee9886ed8c"} Jul 07 13:54:11.831479 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.831475 2576 scope.go:117] "RemoveContainer" containerID="5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c" Jul 07 13:54:11.841196 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.841156 2576 scope.go:117] "RemoveContainer" containerID="5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c" Jul 07 13:54:11.841439 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:54:11.841418 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c\": container with ID starting with 5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c not found: ID does not exist" containerID="5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c" Jul 07 13:54:11.841491 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.841447 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c"} err="failed to get container status \"5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c\": rpc error: code = NotFound desc = could not find container \"5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c\": container with ID starting with 5999ac3c33bdc897595a7d4e875cdd1d2b6c2717a81d3cc0c9c419de1a3f9a4c not found: ID does not exist" Jul 07 13:54:11.856365 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.856340 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg"] Jul 07 13:54:11.861520 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:11.861500 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-987cf14b-openshift-default-84f5755c5-8zddg"] Jul 07 13:54:12.251213 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.251181 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq"] Jul 07 13:54:12.251566 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.251552 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c123df96-03ea-49e1-b7c7-262879ac34bb" containerName="istio-proxy" Jul 07 13:54:12.251616 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.251569 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c123df96-03ea-49e1-b7c7-262879ac34bb" containerName="istio-proxy" Jul 07 13:54:12.251616 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.251583 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af895fe-d12d-4ccb-a81b-695c809eac90" containerName="maas-api" Jul 07 13:54:12.251616 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.251589 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af895fe-d12d-4ccb-a81b-695c809eac90" containerName="maas-api" Jul 07 13:54:12.251719 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.251647 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c123df96-03ea-49e1-b7c7-262879ac34bb" containerName="istio-proxy" Jul 07 13:54:12.251719 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.251657 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af895fe-d12d-4ccb-a81b-695c809eac90" containerName="maas-api" Jul 07 13:54:12.254623 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.254603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.261708 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.261688 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-98f7af91-openshift-default-dockercfg-8bf9c\"" Jul 07 13:54:12.287691 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.287666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq"] Jul 07 13:54:12.384680 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.384651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c956c831-be64-406a-b87c-bb9461762fb1-istio-podinfo\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.384835 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.384687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-credential-socket\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.384835 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.384789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-socket\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.384835 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.384817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-data\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.384942 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.384842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-certs\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.384942 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.384861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-istio-token\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.384942 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.384893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-envoy\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.384942 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.384911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c956c831-be64-406a-b87c-bb9461762fb1-istiod-ca-cert\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.384942 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.384933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq7zb\" (UniqueName: \"kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-kube-api-access-gq7zb\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-socket\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-data\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-certs\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486533 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-istio-token\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486533 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-envoy\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486533 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c956c831-be64-406a-b87c-bb9461762fb1-istiod-ca-cert\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486533 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq7zb\" (UniqueName: \"kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-kube-api-access-gq7zb\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486740 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c956c831-be64-406a-b87c-bb9461762fb1-istio-podinfo\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486740 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-credential-socket\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486740 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486650 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-data\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486740 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486704 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-socket\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486954 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486789 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-certs\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.486954 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.486881 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-credential-socket\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.487074 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.487056 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c956c831-be64-406a-b87c-bb9461762fb1-istiod-ca-cert\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.488725 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.488703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-envoy\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.488875 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.488857 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c956c831-be64-406a-b87c-bb9461762fb1-istio-podinfo\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.496812 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.496789 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-istio-token\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.496944 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.496919 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq7zb\" (UniqueName: \"kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-kube-api-access-gq7zb\") pod \"e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.565370 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.565347 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:12.591309 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.591284 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c123df96-03ea-49e1-b7c7-262879ac34bb" path="/var/lib/kubelet/pods/c123df96-03ea-49e1-b7c7-262879ac34bb/volumes" Jul 07 13:54:12.708242 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.708213 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq"] Jul 07 13:54:12.708968 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:54:12.708946 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc956c831_be64_406a_b87c_bb9461762fb1.slice/crio-77cb1b9e79b2f4778ef742f8d48f16a41c9b8418e52acadc797353c6e16fe31f WatchSource:0}: Error finding container 77cb1b9e79b2f4778ef742f8d48f16a41c9b8418e52acadc797353c6e16fe31f: Status 404 returned error can't find the container with id 77cb1b9e79b2f4778ef742f8d48f16a41c9b8418e52acadc797353c6e16fe31f Jul 07 13:54:12.711013 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.710983 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:54:12.711073 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.711061 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:54:12.711112 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.711092 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:54:12.837652 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.837617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" event={"ID":"c956c831-be64-406a-b87c-bb9461762fb1","Type":"ContainerStarted","Data":"845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67"} Jul 07 13:54:12.837826 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.837684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" event={"ID":"c956c831-be64-406a-b87c-bb9461762fb1","Type":"ContainerStarted","Data":"77cb1b9e79b2f4778ef742f8d48f16a41c9b8418e52acadc797353c6e16fe31f"} Jul 07 13:54:12.873674 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:12.873620 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" podStartSLOduration=0.873601091 podStartE2EDuration="873.601091ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:54:12.873218442 +0000 UTC m=+1586.874459971" watchObservedRunningTime="2026-07-07 13:54:12.873601091 +0000 UTC m=+1586.874842619" Jul 07 13:54:13.551923 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.551887 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh"] Jul 07 13:54:13.554436 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.554415 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:13.558663 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.558641 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-98f7af91-serving-cert\"" Jul 07 13:54:13.565835 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.565818 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:13.571117 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.571095 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:13.571772 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.571752 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh"] Jul 07 13:54:13.697809 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.697777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7969bf-8474-4f0b-a928-9396944018a0-openshift-service-ca\") pod \"maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:13.697809 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.697811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7969bf-8474-4f0b-a928-9396944018a0-maas-api-tls\") pod \"maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:13.698031 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.697848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmknh\" (UniqueName: \"kubernetes.io/projected/be7969bf-8474-4f0b-a928-9396944018a0-kube-api-access-mmknh\") pod \"maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:13.798832 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.798803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmknh\" (UniqueName: \"kubernetes.io/projected/be7969bf-8474-4f0b-a928-9396944018a0-kube-api-access-mmknh\") pod \"maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:13.798980 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.798926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7969bf-8474-4f0b-a928-9396944018a0-openshift-service-ca\") pod \"maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:13.798980 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.798960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7969bf-8474-4f0b-a928-9396944018a0-maas-api-tls\") pod \"maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:13.799096 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:54:13.799063 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-98f7af91-serving-cert: secret "maas-api-e2e-ait-98f7af91-serving-cert" not found Jul 07 13:54:13.799142 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:54:13.799123 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7969bf-8474-4f0b-a928-9396944018a0-maas-api-tls podName:be7969bf-8474-4f0b-a928-9396944018a0 nodeName:}" failed. No retries permitted until 2026-07-07 13:54:14.299105909 +0000 UTC m=+1588.300347414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be7969bf-8474-4f0b-a928-9396944018a0-maas-api-tls") pod "maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" (UID: "be7969bf-8474-4f0b-a928-9396944018a0") : secret "maas-api-e2e-ait-98f7af91-serving-cert" not found Jul 07 13:54:13.799611 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.799588 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7969bf-8474-4f0b-a928-9396944018a0-openshift-service-ca\") pod \"maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:13.813526 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.813474 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmknh\" (UniqueName: \"kubernetes.io/projected/be7969bf-8474-4f0b-a928-9396944018a0-kube-api-access-mmknh\") pod \"maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:13.843217 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.843196 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:13.844139 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:13.844121 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:14.304200 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:14.304152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7969bf-8474-4f0b-a928-9396944018a0-maas-api-tls\") pod \"maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:14.306618 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:14.306592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7969bf-8474-4f0b-a928-9396944018a0-maas-api-tls\") pod \"maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:14.465135 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:14.465104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:14.622317 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:14.622289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh"] Jul 07 13:54:14.623810 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:54:14.623784 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7969bf_8474_4f0b_a928_9396944018a0.slice/crio-fda23e4402c1a13f8a3aa0aa2792636d1144e526374d230644ccdf82dc26e17d WatchSource:0}: Error finding container fda23e4402c1a13f8a3aa0aa2792636d1144e526374d230644ccdf82dc26e17d: Status 404 returned error can't find the container with id fda23e4402c1a13f8a3aa0aa2792636d1144e526374d230644ccdf82dc26e17d Jul 07 13:54:14.850722 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:14.850692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" event={"ID":"be7969bf-8474-4f0b-a928-9396944018a0","Type":"ContainerStarted","Data":"fda23e4402c1a13f8a3aa0aa2792636d1144e526374d230644ccdf82dc26e17d"} Jul 07 13:54:15.857074 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:15.856983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" event={"ID":"be7969bf-8474-4f0b-a928-9396944018a0","Type":"ContainerStarted","Data":"e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091"} Jul 07 13:54:15.894816 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:15.894774 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" podStartSLOduration=2.525382671 podStartE2EDuration="2.894759136s" totalImagesPullingTime="369.376465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:54:15.893329385 +0000 UTC m=+1589.894570911" watchObservedRunningTime="2026-07-07 13:54:15.894759136 +0000 UTC m=+1589.896000699" Jul 07 13:54:16.872938 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:16.872903 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:19.086721 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.086688 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh"] Jul 07 13:54:19.087213 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.086886 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" podUID="be7969bf-8474-4f0b-a928-9396944018a0" containerName="maas-api" containerID="cri-o://e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091" gracePeriod=30 Jul 07 13:54:19.383747 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.383723 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:19.448830 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.448805 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmknh\" (UniqueName: \"kubernetes.io/projected/be7969bf-8474-4f0b-a928-9396944018a0-kube-api-access-mmknh\") pod \"be7969bf-8474-4f0b-a928-9396944018a0\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " Jul 07 13:54:19.449002 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.448858 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7969bf-8474-4f0b-a928-9396944018a0-openshift-service-ca\") pod \"be7969bf-8474-4f0b-a928-9396944018a0\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " Jul 07 13:54:19.449002 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.448893 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7969bf-8474-4f0b-a928-9396944018a0-maas-api-tls\") pod \"be7969bf-8474-4f0b-a928-9396944018a0\" (UID: \"be7969bf-8474-4f0b-a928-9396944018a0\") " Jul 07 13:54:19.449256 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.449231 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7969bf-8474-4f0b-a928-9396944018a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be7969bf-8474-4f0b-a928-9396944018a0" (UID: "be7969bf-8474-4f0b-a928-9396944018a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:54:19.450777 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.450750 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7969bf-8474-4f0b-a928-9396944018a0-kube-api-access-mmknh" (OuterVolumeSpecName: "kube-api-access-mmknh") pod "be7969bf-8474-4f0b-a928-9396944018a0" (UID: "be7969bf-8474-4f0b-a928-9396944018a0"). InnerVolumeSpecName "kube-api-access-mmknh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:54:19.450881 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.450834 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be7969bf-8474-4f0b-a928-9396944018a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be7969bf-8474-4f0b-a928-9396944018a0" (UID: "be7969bf-8474-4f0b-a928-9396944018a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 13:54:19.550574 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.550548 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7969bf-8474-4f0b-a928-9396944018a0-maas-api-tls\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:19.550574 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.550573 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmknh\" (UniqueName: \"kubernetes.io/projected/be7969bf-8474-4f0b-a928-9396944018a0-kube-api-access-mmknh\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:19.550742 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.550583 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7969bf-8474-4f0b-a928-9396944018a0-openshift-service-ca\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:19.892836 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.892800 2576 generic.go:356] "Generic (PLEG): container finished" podID="be7969bf-8474-4f0b-a928-9396944018a0" containerID="e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091" exitCode=0 Jul 07 13:54:19.893022 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.892845 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" Jul 07 13:54:19.893022 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.892887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" event={"ID":"be7969bf-8474-4f0b-a928-9396944018a0","Type":"ContainerDied","Data":"e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091"} Jul 07 13:54:19.893022 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.892944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh" event={"ID":"be7969bf-8474-4f0b-a928-9396944018a0","Type":"ContainerDied","Data":"fda23e4402c1a13f8a3aa0aa2792636d1144e526374d230644ccdf82dc26e17d"} Jul 07 13:54:19.893022 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.892972 2576 scope.go:117] "RemoveContainer" containerID="e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091" Jul 07 13:54:19.901858 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.901843 2576 scope.go:117] "RemoveContainer" containerID="e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091" Jul 07 13:54:19.902095 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:54:19.902079 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091\": container with ID starting with e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091 not found: ID does not exist" containerID="e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091" Jul 07 13:54:19.902151 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.902103 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091"} err="failed to get container status \"e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091\": rpc error: code = NotFound desc = could not find container \"e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091\": container with ID starting with e13471d2304169878be6ac1a32391f1af806865a2bb495b408eaa746ab3dc091 not found: ID does not exist" Jul 07 13:54:19.941866 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.941831 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh"] Jul 07 13:54:19.948807 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:19.948789 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-98f7af91-6fbfc7fc89-4pdbh"] Jul 07 13:54:20.590206 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:20.590159 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be7969bf-8474-4f0b-a928-9396944018a0" path="/var/lib/kubelet/pods/be7969bf-8474-4f0b-a928-9396944018a0/volumes" Jul 07 13:54:23.915838 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:23.915762 2576 generic.go:356] "Generic (PLEG): container finished" podID="c956c831-be64-406a-b87c-bb9461762fb1" containerID="845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67" exitCode=0 Jul 07 13:54:23.916247 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:23.915841 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" event={"ID":"c956c831-be64-406a-b87c-bb9461762fb1","Type":"ContainerDied","Data":"845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67"} Jul 07 13:54:23.916300 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:23.916284 2576 scope.go:117] "RemoveContainer" containerID="845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67" Jul 07 13:54:23.917395 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:23.917364 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:54:23.917497 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:23.917435 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:54:23.917497 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:23.917467 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:54:24.655608 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:24.655578 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq"] Jul 07 13:54:24.922689 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:24.922614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" event={"ID":"c956c831-be64-406a-b87c-bb9461762fb1","Type":"ContainerStarted","Data":"98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f"} Jul 07 13:54:24.923039 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:24.922710 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" podUID="c956c831-be64-406a-b87c-bb9461762fb1" containerName="istio-proxy" containerID="cri-o://98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f" gracePeriod=30 Jul 07 13:54:28.843044 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:28.843014 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:30.178924 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.178900 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:30.349597 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.349559 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-credential-socket\") pod \"c956c831-be64-406a-b87c-bb9461762fb1\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " Jul 07 13:54:30.349790 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.349624 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-envoy\") pod \"c956c831-be64-406a-b87c-bb9461762fb1\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " Jul 07 13:54:30.349790 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.349658 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-istio-token\") pod \"c956c831-be64-406a-b87c-bb9461762fb1\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " Jul 07 13:54:30.349790 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.349674 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c956c831-be64-406a-b87c-bb9461762fb1-istiod-ca-cert\") pod \"c956c831-be64-406a-b87c-bb9461762fb1\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " Jul 07 13:54:30.349790 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.349708 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-data\") pod \"c956c831-be64-406a-b87c-bb9461762fb1\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " Jul 07 13:54:30.349790 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.349732 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq7zb\" (UniqueName: \"kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-kube-api-access-gq7zb\") pod \"c956c831-be64-406a-b87c-bb9461762fb1\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " Jul 07 13:54:30.350047 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.349865 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c956c831-be64-406a-b87c-bb9461762fb1" (UID: "c956c831-be64-406a-b87c-bb9461762fb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:30.350047 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.349872 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-certs\") pod \"c956c831-be64-406a-b87c-bb9461762fb1\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " Jul 07 13:54:30.350047 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.350008 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-socket\") pod \"c956c831-be64-406a-b87c-bb9461762fb1\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " Jul 07 13:54:30.350047 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.350041 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c956c831-be64-406a-b87c-bb9461762fb1-istio-podinfo\") pod \"c956c831-be64-406a-b87c-bb9461762fb1\" (UID: \"c956c831-be64-406a-b87c-bb9461762fb1\") " Jul 07 13:54:30.350647 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.350084 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c956c831-be64-406a-b87c-bb9461762fb1" (UID: "c956c831-be64-406a-b87c-bb9461762fb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:30.350647 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.350134 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c956c831-be64-406a-b87c-bb9461762fb1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c956c831-be64-406a-b87c-bb9461762fb1" (UID: "c956c831-be64-406a-b87c-bb9461762fb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:54:30.350647 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.350301 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c956c831-be64-406a-b87c-bb9461762fb1" (UID: "c956c831-be64-406a-b87c-bb9461762fb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:30.350647 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.350460 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:30.350647 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.350475 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c956c831-be64-406a-b87c-bb9461762fb1-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:30.350647 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.350485 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:30.350647 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.350494 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:30.350647 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.350639 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-data" (OuterVolumeSpecName: "istio-data") pod "c956c831-be64-406a-b87c-bb9461762fb1" (UID: "c956c831-be64-406a-b87c-bb9461762fb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:30.351966 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.351939 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c956c831-be64-406a-b87c-bb9461762fb1" (UID: "c956c831-be64-406a-b87c-bb9461762fb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:30.352077 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.352047 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c956c831-be64-406a-b87c-bb9461762fb1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c956c831-be64-406a-b87c-bb9461762fb1" (UID: "c956c831-be64-406a-b87c-bb9461762fb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 13:54:30.352366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.352346 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-istio-token" (OuterVolumeSpecName: "istio-token") pod "c956c831-be64-406a-b87c-bb9461762fb1" (UID: "c956c831-be64-406a-b87c-bb9461762fb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:54:30.352434 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.352359 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-kube-api-access-gq7zb" (OuterVolumeSpecName: "kube-api-access-gq7zb") pod "c956c831-be64-406a-b87c-bb9461762fb1" (UID: "c956c831-be64-406a-b87c-bb9461762fb1"). InnerVolumeSpecName "kube-api-access-gq7zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:54:30.451270 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.451237 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:30.451270 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.451266 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq7zb\" (UniqueName: \"kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-kube-api-access-gq7zb\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:30.451270 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.451277 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c956c831-be64-406a-b87c-bb9461762fb1-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:30.451492 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.451286 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c956c831-be64-406a-b87c-bb9461762fb1-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:30.451492 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.451294 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c956c831-be64-406a-b87c-bb9461762fb1-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:30.953192 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.953149 2576 generic.go:356] "Generic (PLEG): container finished" podID="c956c831-be64-406a-b87c-bb9461762fb1" containerID="98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f" exitCode=0 Jul 07 13:54:30.953399 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.953195 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" event={"ID":"c956c831-be64-406a-b87c-bb9461762fb1","Type":"ContainerDied","Data":"98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f"} Jul 07 13:54:30.953399 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.953210 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" Jul 07 13:54:30.953399 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.953258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq" event={"ID":"c956c831-be64-406a-b87c-bb9461762fb1","Type":"ContainerDied","Data":"77cb1b9e79b2f4778ef742f8d48f16a41c9b8418e52acadc797353c6e16fe31f"} Jul 07 13:54:30.953399 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.953290 2576 scope.go:117] "RemoveContainer" containerID="98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f" Jul 07 13:54:30.962327 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.962310 2576 scope.go:117] "RemoveContainer" containerID="845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67" Jul 07 13:54:30.969870 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.969853 2576 scope.go:117] "RemoveContainer" containerID="98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f" Jul 07 13:54:30.970129 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:54:30.970111 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f\": container with ID starting with 98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f not found: ID does not exist" containerID="98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f" Jul 07 13:54:30.970216 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.970138 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f"} err="failed to get container status \"98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f\": rpc error: code = NotFound desc = could not find container \"98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f\": container with ID starting with 98b578bad207f43338e811331d0669968f3944c7423605e93d50de9d6a54919f not found: ID does not exist" Jul 07 13:54:30.970216 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.970154 2576 scope.go:117] "RemoveContainer" containerID="845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67" Jul 07 13:54:30.970437 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:54:30.970420 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67\": container with ID starting with 845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67 not found: ID does not exist" containerID="845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67" Jul 07 13:54:30.970479 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.970450 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67"} err="failed to get container status \"845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67\": rpc error: code = NotFound desc = could not find container \"845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67\": container with ID starting with 845a6df7bbeee7a46dc0a7162b15058e531e12326bd610639ef07fbf1778eb67 not found: ID does not exist" Jul 07 13:54:30.982565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.982545 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq"] Jul 07 13:54:30.990460 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:30.990436 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-98f7af91-openshift-default-664dc76b85-4hlxq"] Jul 07 13:54:31.696024 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.695996 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r"] Jul 07 13:54:31.696508 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.696491 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7969bf-8474-4f0b-a928-9396944018a0" containerName="maas-api" Jul 07 13:54:31.696565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.696510 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7969bf-8474-4f0b-a928-9396944018a0" containerName="maas-api" Jul 07 13:54:31.696565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.696533 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c956c831-be64-406a-b87c-bb9461762fb1" containerName="istio-proxy" Jul 07 13:54:31.696565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.696539 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c956c831-be64-406a-b87c-bb9461762fb1" containerName="istio-proxy" Jul 07 13:54:31.696565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.696551 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c956c831-be64-406a-b87c-bb9461762fb1" containerName="istio-proxy" Jul 07 13:54:31.696565 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.696556 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c956c831-be64-406a-b87c-bb9461762fb1" containerName="istio-proxy" Jul 07 13:54:31.696729 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.696608 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7969bf-8474-4f0b-a928-9396944018a0" containerName="maas-api" Jul 07 13:54:31.696729 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.696616 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c956c831-be64-406a-b87c-bb9461762fb1" containerName="istio-proxy" Jul 07 13:54:31.699634 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.699612 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.703556 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.703533 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c1d14ac0-openshift-default-dockercfg-6xnzd\"" Jul 07 13:54:31.719747 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.719726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r"] Jul 07 13:54:31.863105 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.863072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15417699-783f-4a10-b956-e7fb50f684bf-istiod-ca-cert\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.863105 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.863112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-data\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.863361 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.863149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-envoy\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.863361 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.863282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-istio-token\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.863361 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.863316 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cskkt\" (UniqueName: \"kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-kube-api-access-cskkt\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.863361 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.863339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-certs\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.863510 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.863393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15417699-783f-4a10-b956-e7fb50f684bf-istio-podinfo\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.863510 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.863446 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-socket\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.863510 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.863479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-credential-socket\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.964966 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.964880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-envoy\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.964966 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.964933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-istio-token\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.964966 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.964961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cskkt\" (UniqueName: \"kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-kube-api-access-cskkt\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.964988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-certs\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.965013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15417699-783f-4a10-b956-e7fb50f684bf-istio-podinfo\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.965046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-socket\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.965083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-credential-socket\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.965136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15417699-783f-4a10-b956-e7fb50f684bf-istiod-ca-cert\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.965208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-data\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965601 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.965416 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-certs\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965601 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.965474 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-socket\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965601 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.965515 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-credential-socket\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965601 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.965577 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-data\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.965867 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.965848 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15417699-783f-4a10-b956-e7fb50f684bf-istiod-ca-cert\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.967047 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.967029 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-envoy\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.967456 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.967434 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15417699-783f-4a10-b956-e7fb50f684bf-istio-podinfo\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.973771 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.973747 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-istio-token\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:31.974155 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:31.974136 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cskkt\" (UniqueName: \"kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-kube-api-access-cskkt\") pod \"e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:32.011755 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:32.011734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:32.152105 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:32.152067 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r"] Jul 07 13:54:32.153035 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:54:32.153007 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15417699_783f_4a10_b956_e7fb50f684bf.slice/crio-199942375d62f98f2a3a7ac55e0ea1eaeedf93cff724e9c6b709f955676a03a2 WatchSource:0}: Error finding container 199942375d62f98f2a3a7ac55e0ea1eaeedf93cff724e9c6b709f955676a03a2: Status 404 returned error can't find the container with id 199942375d62f98f2a3a7ac55e0ea1eaeedf93cff724e9c6b709f955676a03a2 Jul 07 13:54:32.155258 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:32.155213 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:54:32.155340 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:32.155298 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:54:32.155379 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:32.155337 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:54:32.590423 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:32.590383 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c956c831-be64-406a-b87c-bb9461762fb1" path="/var/lib/kubelet/pods/c956c831-be64-406a-b87c-bb9461762fb1/volumes" Jul 07 13:54:32.965321 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:32.965236 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" event={"ID":"15417699-783f-4a10-b956-e7fb50f684bf","Type":"ContainerStarted","Data":"9f0126505eb924e6697f81ec3cab45f1c594819bb4e70bb8da9bcb113fccdf67"} Jul 07 13:54:32.965321 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:32.965293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" event={"ID":"15417699-783f-4a10-b956-e7fb50f684bf","Type":"ContainerStarted","Data":"199942375d62f98f2a3a7ac55e0ea1eaeedf93cff724e9c6b709f955676a03a2"} Jul 07 13:54:33.012655 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.012622 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:33.017495 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.017465 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:33.043197 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.043133 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" podStartSLOduration=2.043117036 podStartE2EDuration="2.043117036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:54:33.041944999 +0000 UTC m=+1607.043186525" watchObservedRunningTime="2026-07-07 13:54:33.043117036 +0000 UTC m=+1607.044358614" Jul 07 13:54:33.140815 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.140779 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5"] Jul 07 13:54:33.141283 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.141264 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c956c831-be64-406a-b87c-bb9461762fb1" containerName="istio-proxy" Jul 07 13:54:33.143431 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.143414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.147422 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.147405 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-c1d14ac0-serving-cert\"" Jul 07 13:54:33.172808 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.172784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5"] Jul 07 13:54:33.278146 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.278114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d084b08f-00d9-4c55-b447-5178f7ff584a-maas-api-tls\") pod \"maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.278309 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.278245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5v7\" (UniqueName: \"kubernetes.io/projected/d084b08f-00d9-4c55-b447-5178f7ff584a-kube-api-access-tb5v7\") pod \"maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.278370 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.278304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d084b08f-00d9-4c55-b447-5178f7ff584a-openshift-service-ca\") pod \"maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.379354 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.379296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5v7\" (UniqueName: \"kubernetes.io/projected/d084b08f-00d9-4c55-b447-5178f7ff584a-kube-api-access-tb5v7\") pod \"maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.379556 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.379366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d084b08f-00d9-4c55-b447-5178f7ff584a-openshift-service-ca\") pod \"maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.379556 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.379435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d084b08f-00d9-4c55-b447-5178f7ff584a-maas-api-tls\") pod \"maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.380526 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.380491 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d084b08f-00d9-4c55-b447-5178f7ff584a-openshift-service-ca\") pod \"maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.382545 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.382517 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d084b08f-00d9-4c55-b447-5178f7ff584a-maas-api-tls\") pod \"maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.391491 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.391468 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5v7\" (UniqueName: \"kubernetes.io/projected/d084b08f-00d9-4c55-b447-5178f7ff584a-kube-api-access-tb5v7\") pod \"maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.454608 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.454581 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:33.801135 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.801109 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5"] Jul 07 13:54:33.802846 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:54:33.802823 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd084b08f_00d9_4c55_b447_5178f7ff584a.slice/crio-ac37c06e562ded7eb6c84f46e122209bccedafdf971f783c02e35b57a6ce31af WatchSource:0}: Error finding container ac37c06e562ded7eb6c84f46e122209bccedafdf971f783c02e35b57a6ce31af: Status 404 returned error can't find the container with id ac37c06e562ded7eb6c84f46e122209bccedafdf971f783c02e35b57a6ce31af Jul 07 13:54:33.977521 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.977414 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" event={"ID":"d084b08f-00d9-4c55-b447-5178f7ff584a","Type":"ContainerStarted","Data":"ac37c06e562ded7eb6c84f46e122209bccedafdf971f783c02e35b57a6ce31af"} Jul 07 13:54:33.977972 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.977740 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:33.979031 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:33.979005 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:34.983525 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:34.983489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" event={"ID":"d084b08f-00d9-4c55-b447-5178f7ff584a","Type":"ContainerStarted","Data":"e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc"} Jul 07 13:54:35.014591 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:35.014544 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" podStartSLOduration=1.668458141 podStartE2EDuration="2.014528133s" totalImagesPullingTime="346.069992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:54:35.01108938 +0000 UTC m=+1609.012330932" watchObservedRunningTime="2026-07-07 13:54:35.014528133 +0000 UTC m=+1609.015769660" Jul 07 13:54:35.993199 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:35.993158 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:37.470327 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:37.470096 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r"] Jul 07 13:54:37.471233 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:37.471198 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" podUID="15417699-783f-4a10-b956-e7fb50f684bf" containerName="istio-proxy" containerID="cri-o://9f0126505eb924e6697f81ec3cab45f1c594819bb4e70bb8da9bcb113fccdf67" gracePeriod=30 Jul 07 13:54:37.781674 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:37.781641 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5"] Jul 07 13:54:38.006624 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.006574 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" podUID="d084b08f-00d9-4c55-b447-5178f7ff584a" containerName="maas-api" containerID="cri-o://e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc" gracePeriod=30 Jul 07 13:54:38.297869 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.297845 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:38.323499 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.323477 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d084b08f-00d9-4c55-b447-5178f7ff584a-maas-api-tls\") pod \"d084b08f-00d9-4c55-b447-5178f7ff584a\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " Jul 07 13:54:38.323650 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.323507 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d084b08f-00d9-4c55-b447-5178f7ff584a-openshift-service-ca\") pod \"d084b08f-00d9-4c55-b447-5178f7ff584a\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " Jul 07 13:54:38.323650 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.323562 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb5v7\" (UniqueName: \"kubernetes.io/projected/d084b08f-00d9-4c55-b447-5178f7ff584a-kube-api-access-tb5v7\") pod \"d084b08f-00d9-4c55-b447-5178f7ff584a\" (UID: \"d084b08f-00d9-4c55-b447-5178f7ff584a\") " Jul 07 13:54:38.323951 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.323925 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d084b08f-00d9-4c55-b447-5178f7ff584a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d084b08f-00d9-4c55-b447-5178f7ff584a" (UID: "d084b08f-00d9-4c55-b447-5178f7ff584a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:54:38.326008 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.325985 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d084b08f-00d9-4c55-b447-5178f7ff584a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d084b08f-00d9-4c55-b447-5178f7ff584a" (UID: "d084b08f-00d9-4c55-b447-5178f7ff584a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 13:54:38.326133 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.326019 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d084b08f-00d9-4c55-b447-5178f7ff584a-kube-api-access-tb5v7" (OuterVolumeSpecName: "kube-api-access-tb5v7") pod "d084b08f-00d9-4c55-b447-5178f7ff584a" (UID: "d084b08f-00d9-4c55-b447-5178f7ff584a"). InnerVolumeSpecName "kube-api-access-tb5v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:54:38.424716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.424636 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb5v7\" (UniqueName: \"kubernetes.io/projected/d084b08f-00d9-4c55-b447-5178f7ff584a-kube-api-access-tb5v7\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:38.424716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.424662 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d084b08f-00d9-4c55-b447-5178f7ff584a-maas-api-tls\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:38.424716 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:38.424673 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d084b08f-00d9-4c55-b447-5178f7ff584a-openshift-service-ca\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:39.014321 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:39.014278 2576 generic.go:356] "Generic (PLEG): container finished" podID="d084b08f-00d9-4c55-b447-5178f7ff584a" containerID="e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc" exitCode=0 Jul 07 13:54:39.014817 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:39.014339 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" Jul 07 13:54:39.014817 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:39.014367 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" event={"ID":"d084b08f-00d9-4c55-b447-5178f7ff584a","Type":"ContainerDied","Data":"e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc"} Jul 07 13:54:39.014817 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:39.014434 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5" event={"ID":"d084b08f-00d9-4c55-b447-5178f7ff584a","Type":"ContainerDied","Data":"ac37c06e562ded7eb6c84f46e122209bccedafdf971f783c02e35b57a6ce31af"} Jul 07 13:54:39.014817 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:39.014460 2576 scope.go:117] "RemoveContainer" containerID="e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc" Jul 07 13:54:39.023496 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:39.023480 2576 scope.go:117] "RemoveContainer" containerID="e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc" Jul 07 13:54:39.023757 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:54:39.023736 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc\": container with ID starting with e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc not found: ID does not exist" containerID="e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc" Jul 07 13:54:39.023856 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:39.023764 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc"} err="failed to get container status \"e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc\": rpc error: code = NotFound desc = could not find container \"e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc\": container with ID starting with e26af4c5536d9f79c599a4b94e721b510c0d7ff7e25afbddcdbf82258c6e33dc not found: ID does not exist" Jul 07 13:54:39.036474 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:39.036452 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5"] Jul 07 13:54:39.044960 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:39.044937 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-c1d14ac0-84dd9b8594-4p6q5"] Jul 07 13:54:40.590584 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:40.590549 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d084b08f-00d9-4c55-b447-5178f7ff584a" path="/var/lib/kubelet/pods/d084b08f-00d9-4c55-b447-5178f7ff584a/volumes" Jul 07 13:54:43.037479 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.037444 2576 generic.go:356] "Generic (PLEG): container finished" podID="15417699-783f-4a10-b956-e7fb50f684bf" containerID="9f0126505eb924e6697f81ec3cab45f1c594819bb4e70bb8da9bcb113fccdf67" exitCode=0 Jul 07 13:54:43.037867 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.037511 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" event={"ID":"15417699-783f-4a10-b956-e7fb50f684bf","Type":"ContainerDied","Data":"9f0126505eb924e6697f81ec3cab45f1c594819bb4e70bb8da9bcb113fccdf67"} Jul 07 13:54:43.127849 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.127828 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:43.171317 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171294 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-istio-token\") pod \"15417699-783f-4a10-b956-e7fb50f684bf\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " Jul 07 13:54:43.171483 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171321 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15417699-783f-4a10-b956-e7fb50f684bf-istio-podinfo\") pod \"15417699-783f-4a10-b956-e7fb50f684bf\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " Jul 07 13:54:43.171483 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171342 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-socket\") pod \"15417699-783f-4a10-b956-e7fb50f684bf\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " Jul 07 13:54:43.171483 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171367 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cskkt\" (UniqueName: \"kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-kube-api-access-cskkt\") pod \"15417699-783f-4a10-b956-e7fb50f684bf\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " Jul 07 13:54:43.171483 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171433 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-credential-socket\") pod \"15417699-783f-4a10-b956-e7fb50f684bf\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " Jul 07 13:54:43.171719 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171495 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-certs\") pod \"15417699-783f-4a10-b956-e7fb50f684bf\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " Jul 07 13:54:43.171719 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171523 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15417699-783f-4a10-b956-e7fb50f684bf-istiod-ca-cert\") pod \"15417699-783f-4a10-b956-e7fb50f684bf\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " Jul 07 13:54:43.171719 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171579 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-envoy\") pod \"15417699-783f-4a10-b956-e7fb50f684bf\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " Jul 07 13:54:43.171719 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171613 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15417699-783f-4a10-b956-e7fb50f684bf" (UID: "15417699-783f-4a10-b956-e7fb50f684bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:43.171719 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171648 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-data\") pod \"15417699-783f-4a10-b956-e7fb50f684bf\" (UID: \"15417699-783f-4a10-b956-e7fb50f684bf\") " Jul 07 13:54:43.171986 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171916 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15417699-783f-4a10-b956-e7fb50f684bf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15417699-783f-4a10-b956-e7fb50f684bf" (UID: "15417699-783f-4a10-b956-e7fb50f684bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:54:43.171986 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.171936 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:43.172340 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.172274 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15417699-783f-4a10-b956-e7fb50f684bf" (UID: "15417699-783f-4a10-b956-e7fb50f684bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:43.172648 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.172621 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15417699-783f-4a10-b956-e7fb50f684bf" (UID: "15417699-783f-4a10-b956-e7fb50f684bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:43.172806 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.172777 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-data" (OuterVolumeSpecName: "istio-data") pod "15417699-783f-4a10-b956-e7fb50f684bf" (UID: "15417699-783f-4a10-b956-e7fb50f684bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:43.174259 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.173792 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15417699-783f-4a10-b956-e7fb50f684bf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15417699-783f-4a10-b956-e7fb50f684bf" (UID: "15417699-783f-4a10-b956-e7fb50f684bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 13:54:43.174259 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.174041 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-kube-api-access-cskkt" (OuterVolumeSpecName: "kube-api-access-cskkt") pod "15417699-783f-4a10-b956-e7fb50f684bf" (UID: "15417699-783f-4a10-b956-e7fb50f684bf"). InnerVolumeSpecName "kube-api-access-cskkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:54:43.174259 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.174079 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-istio-token" (OuterVolumeSpecName: "istio-token") pod "15417699-783f-4a10-b956-e7fb50f684bf" (UID: "15417699-783f-4a10-b956-e7fb50f684bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:54:43.174481 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.174299 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15417699-783f-4a10-b956-e7fb50f684bf" (UID: "15417699-783f-4a10-b956-e7fb50f684bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:54:43.272638 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.272610 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cskkt\" (UniqueName: \"kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-kube-api-access-cskkt\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:43.272638 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.272637 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:43.272814 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.272647 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:43.272814 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.272656 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15417699-783f-4a10-b956-e7fb50f684bf-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:43.272814 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.272664 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:43.272814 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.272672 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15417699-783f-4a10-b956-e7fb50f684bf-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:43.272814 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.272679 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15417699-783f-4a10-b956-e7fb50f684bf-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:43.272814 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:43.272687 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15417699-783f-4a10-b956-e7fb50f684bf-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:54:44.044594 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:44.044555 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" event={"ID":"15417699-783f-4a10-b956-e7fb50f684bf","Type":"ContainerDied","Data":"199942375d62f98f2a3a7ac55e0ea1eaeedf93cff724e9c6b709f955676a03a2"} Jul 07 13:54:44.044594 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:44.044585 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r" Jul 07 13:54:44.045100 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:44.044630 2576 scope.go:117] "RemoveContainer" containerID="9f0126505eb924e6697f81ec3cab45f1c594819bb4e70bb8da9bcb113fccdf67" Jul 07 13:54:44.068153 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:44.068122 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r"] Jul 07 13:54:44.070955 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:44.070931 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c1d14ac0-openshift-default-6b7c5c8dd7-bxg6r"] Jul 07 13:54:44.590270 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:54:44.590232 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15417699-783f-4a10-b956-e7fb50f684bf" path="/var/lib/kubelet/pods/15417699-783f-4a10-b956-e7fb50f684bf/volumes" Jul 07 13:56:26.398154 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.398118 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m"] Jul 07 13:56:26.398625 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.398503 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d084b08f-00d9-4c55-b447-5178f7ff584a" containerName="maas-api" Jul 07 13:56:26.398625 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.398514 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d084b08f-00d9-4c55-b447-5178f7ff584a" containerName="maas-api" Jul 07 13:56:26.398625 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.398528 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15417699-783f-4a10-b956-e7fb50f684bf" containerName="istio-proxy" Jul 07 13:56:26.398625 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.398534 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15417699-783f-4a10-b956-e7fb50f684bf" containerName="istio-proxy" Jul 07 13:56:26.398625 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.398590 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15417699-783f-4a10-b956-e7fb50f684bf" containerName="istio-proxy" Jul 07 13:56:26.398625 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.398598 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d084b08f-00d9-4c55-b447-5178f7ff584a" containerName="maas-api" Jul 07 13:56:26.401669 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.401654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.405183 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.405134 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0e8fd7f6-openshift-default-dockercfg-27vfr\"" Jul 07 13:56:26.452998 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.452959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-credential-socket\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.452998 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.452998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-socket\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.453294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.453016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-certs\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.453294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.453041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-token\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.453294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.453130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-envoy\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.453294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.453196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istiod-ca-cert\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.453294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.453244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-podinfo\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.453486 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.453329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-data\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.453486 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.453369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ptlj\" (UniqueName: \"kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-kube-api-access-9ptlj\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.460621 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.460596 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m"] Jul 07 13:56:26.554001 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.553969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-data\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ptlj\" (UniqueName: \"kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-kube-api-access-9ptlj\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-credential-socket\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-socket\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-certs\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-token\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-envoy\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554204 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istiod-ca-cert\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554725 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-podinfo\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554725 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554419 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-data\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554725 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554476 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-credential-socket\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554725 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-socket\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.554947 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.554883 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-certs\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.555092 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.555069 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istiod-ca-cert\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.556625 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.556605 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-envoy\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.556714 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.556702 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-podinfo\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.574530 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.574499 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ptlj\" (UniqueName: \"kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-kube-api-access-9ptlj\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.575278 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.575260 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-token\") pod \"e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.718273 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.718159 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:26.863763 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.863736 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m"] Jul 07 13:56:26.865396 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:56:26.865372 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbd6e3c_fca8_40bb_a340_9b6433c0b077.slice/crio-ecef18f9fe6d9591c1d1e8e7b4a99d6faadcced1d5f6f8ac983a4aa128c5f991 WatchSource:0}: Error finding container ecef18f9fe6d9591c1d1e8e7b4a99d6faadcced1d5f6f8ac983a4aa128c5f991: Status 404 returned error can't find the container with id ecef18f9fe6d9591c1d1e8e7b4a99d6faadcced1d5f6f8ac983a4aa128c5f991 Jul 07 13:56:26.867477 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.867444 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:56:26.867570 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.867517 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:56:26.867633 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:26.867567 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:56:27.547296 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:27.547247 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" event={"ID":"0cbd6e3c-fca8-40bb-a340-9b6433c0b077","Type":"ContainerStarted","Data":"3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0"} Jul 07 13:56:27.547689 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:27.547318 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" event={"ID":"0cbd6e3c-fca8-40bb-a340-9b6433c0b077","Type":"ContainerStarted","Data":"ecef18f9fe6d9591c1d1e8e7b4a99d6faadcced1d5f6f8ac983a4aa128c5f991"} Jul 07 13:56:27.575246 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:27.575199 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" podStartSLOduration=1.575184393 podStartE2EDuration="1.575184393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:56:27.574538085 +0000 UTC m=+1721.575779612" watchObservedRunningTime="2026-07-07 13:56:27.575184393 +0000 UTC m=+1721.576425914" Jul 07 13:56:27.718887 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:27.718856 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:27.723875 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:27.723850 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:28.552685 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:28.552653 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:28.553697 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:28.553675 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:29.724277 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.724246 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs"] Jul 07 13:56:29.727678 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.727656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:29.731145 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.731120 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-0e8fd7f6-serving-cert\"" Jul 07 13:56:29.753603 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.753571 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs"] Jul 07 13:56:29.784272 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.784245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/120136d0-36d6-45fe-b1a6-4dc551c6531c-openshift-service-ca\") pod \"maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:29.784402 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.784284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120136d0-36d6-45fe-b1a6-4dc551c6531c-maas-api-tls\") pod \"maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:29.784402 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.784349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7j6r\" (UniqueName: \"kubernetes.io/projected/120136d0-36d6-45fe-b1a6-4dc551c6531c-kube-api-access-x7j6r\") pod \"maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:29.885609 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.885581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/120136d0-36d6-45fe-b1a6-4dc551c6531c-openshift-service-ca\") pod \"maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:29.885783 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.885622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120136d0-36d6-45fe-b1a6-4dc551c6531c-maas-api-tls\") pod \"maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:29.885783 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:56:29.885721 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-0e8fd7f6-serving-cert: secret "maas-api-e2e-mt-0e8fd7f6-serving-cert" not found Jul 07 13:56:29.885783 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.885752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7j6r\" (UniqueName: \"kubernetes.io/projected/120136d0-36d6-45fe-b1a6-4dc551c6531c-kube-api-access-x7j6r\") pod \"maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:29.885783 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:56:29.885773 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/120136d0-36d6-45fe-b1a6-4dc551c6531c-maas-api-tls podName:120136d0-36d6-45fe-b1a6-4dc551c6531c nodeName:}" failed. No retries permitted until 2026-07-07 13:56:30.385757772 +0000 UTC m=+1724.386999277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/120136d0-36d6-45fe-b1a6-4dc551c6531c-maas-api-tls") pod "maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" (UID: "120136d0-36d6-45fe-b1a6-4dc551c6531c") : secret "maas-api-e2e-mt-0e8fd7f6-serving-cert" not found Jul 07 13:56:29.886284 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.886267 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/120136d0-36d6-45fe-b1a6-4dc551c6531c-openshift-service-ca\") pod \"maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:29.900017 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:29.899994 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7j6r\" (UniqueName: \"kubernetes.io/projected/120136d0-36d6-45fe-b1a6-4dc551c6531c-kube-api-access-x7j6r\") pod \"maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:30.391999 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:30.391961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120136d0-36d6-45fe-b1a6-4dc551c6531c-maas-api-tls\") pod \"maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:30.394416 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:30.394388 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120136d0-36d6-45fe-b1a6-4dc551c6531c-maas-api-tls\") pod \"maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:30.641142 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:30.641114 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:30.790124 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:30.790099 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs"] Jul 07 13:56:30.792729 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:56:30.792702 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120136d0_36d6_45fe_b1a6_4dc551c6531c.slice/crio-afdbd01e0f72fae3cd583d94f7a04c702a25593dd9ce7ac5a8d542694a18232b WatchSource:0}: Error finding container afdbd01e0f72fae3cd583d94f7a04c702a25593dd9ce7ac5a8d542694a18232b: Status 404 returned error can't find the container with id afdbd01e0f72fae3cd583d94f7a04c702a25593dd9ce7ac5a8d542694a18232b Jul 07 13:56:31.571264 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:31.571136 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" event={"ID":"120136d0-36d6-45fe-b1a6-4dc551c6531c","Type":"ContainerStarted","Data":"9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad"} Jul 07 13:56:31.571264 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:31.571222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" event={"ID":"120136d0-36d6-45fe-b1a6-4dc551c6531c","Type":"ContainerStarted","Data":"afdbd01e0f72fae3cd583d94f7a04c702a25593dd9ce7ac5a8d542694a18232b"} Jul 07 13:56:31.571264 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:31.571254 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:31.597611 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:31.597566 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" podStartSLOduration=2.073607354 podStartE2EDuration="2.597552065s" totalImagesPullingTime="523.944711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:56:31.595613203 +0000 UTC m=+1725.596854730" watchObservedRunningTime="2026-07-07 13:56:31.597552065 +0000 UTC m=+1725.598793592" Jul 07 13:56:37.583214 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:37.583183 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:43.191830 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.191797 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs"] Jul 07 13:56:43.192289 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.192023 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" podUID="120136d0-36d6-45fe-b1a6-4dc551c6531c" containerName="maas-api" containerID="cri-o://9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad" gracePeriod=30 Jul 07 13:56:43.449014 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.448954 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:43.514997 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.514968 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7j6r\" (UniqueName: \"kubernetes.io/projected/120136d0-36d6-45fe-b1a6-4dc551c6531c-kube-api-access-x7j6r\") pod \"120136d0-36d6-45fe-b1a6-4dc551c6531c\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " Jul 07 13:56:43.516394 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.515009 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/120136d0-36d6-45fe-b1a6-4dc551c6531c-openshift-service-ca\") pod \"120136d0-36d6-45fe-b1a6-4dc551c6531c\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " Jul 07 13:56:43.516394 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.515028 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120136d0-36d6-45fe-b1a6-4dc551c6531c-maas-api-tls\") pod \"120136d0-36d6-45fe-b1a6-4dc551c6531c\" (UID: \"120136d0-36d6-45fe-b1a6-4dc551c6531c\") " Jul 07 13:56:43.516394 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.515421 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120136d0-36d6-45fe-b1a6-4dc551c6531c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "120136d0-36d6-45fe-b1a6-4dc551c6531c" (UID: "120136d0-36d6-45fe-b1a6-4dc551c6531c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:56:43.517085 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.517054 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120136d0-36d6-45fe-b1a6-4dc551c6531c-kube-api-access-x7j6r" (OuterVolumeSpecName: "kube-api-access-x7j6r") pod "120136d0-36d6-45fe-b1a6-4dc551c6531c" (UID: "120136d0-36d6-45fe-b1a6-4dc551c6531c"). InnerVolumeSpecName "kube-api-access-x7j6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:56:43.517085 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.517068 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/120136d0-36d6-45fe-b1a6-4dc551c6531c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "120136d0-36d6-45fe-b1a6-4dc551c6531c" (UID: "120136d0-36d6-45fe-b1a6-4dc551c6531c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 13:56:43.554516 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.554488 2576 patch_prober.go:28] interesting pod/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:60736->10.134.0.55:15021: read: connection reset by peer" start-of-body= Jul 07 13:56:43.554634 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.554547 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:60736->10.134.0.55:15021: read: connection reset by peer" Jul 07 13:56:43.615921 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.615894 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7j6r\" (UniqueName: \"kubernetes.io/projected/120136d0-36d6-45fe-b1a6-4dc551c6531c-kube-api-access-x7j6r\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:43.615921 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.615919 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/120136d0-36d6-45fe-b1a6-4dc551c6531c-openshift-service-ca\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:43.616071 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.615929 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120136d0-36d6-45fe-b1a6-4dc551c6531c-maas-api-tls\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:43.634252 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.634220 2576 generic.go:356] "Generic (PLEG): container finished" podID="120136d0-36d6-45fe-b1a6-4dc551c6531c" containerID="9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad" exitCode=0 Jul 07 13:56:43.634350 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.634274 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" Jul 07 13:56:43.634350 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.634306 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" event={"ID":"120136d0-36d6-45fe-b1a6-4dc551c6531c","Type":"ContainerDied","Data":"9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad"} Jul 07 13:56:43.634438 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.634361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs" event={"ID":"120136d0-36d6-45fe-b1a6-4dc551c6531c","Type":"ContainerDied","Data":"afdbd01e0f72fae3cd583d94f7a04c702a25593dd9ce7ac5a8d542694a18232b"} Jul 07 13:56:43.634438 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.634393 2576 scope.go:117] "RemoveContainer" containerID="9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad" Jul 07 13:56:43.636033 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.636012 2576 generic.go:356] "Generic (PLEG): container finished" podID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerID="3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0" exitCode=0 Jul 07 13:56:43.636132 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.636102 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" event={"ID":"0cbd6e3c-fca8-40bb-a340-9b6433c0b077","Type":"ContainerDied","Data":"3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0"} Jul 07 13:56:43.636531 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.636511 2576 scope.go:117] "RemoveContainer" containerID="3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0" Jul 07 13:56:43.637528 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.637495 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:56:43.637619 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.637572 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:56:43.637619 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.637611 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:56:43.646113 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.646097 2576 scope.go:117] "RemoveContainer" containerID="9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad" Jul 07 13:56:43.646622 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:56:43.646603 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad\": container with ID starting with 9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad not found: ID does not exist" containerID="9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad" Jul 07 13:56:43.646733 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.646629 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad"} err="failed to get container status \"9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad\": rpc error: code = NotFound desc = could not find container \"9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad\": container with ID starting with 9df848e6f01287a6c4add8c823b4818260d601ba2744817b52f165c1392f85ad not found: ID does not exist" Jul 07 13:56:43.755549 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.755519 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs"] Jul 07 13:56:43.788158 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:43.788130 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-0e8fd7f6-9464944b5-hl2gs"] Jul 07 13:56:44.590345 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:44.590314 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="120136d0-36d6-45fe-b1a6-4dc551c6531c" path="/var/lib/kubelet/pods/120136d0-36d6-45fe-b1a6-4dc551c6531c/volumes" Jul 07 13:56:44.643151 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:44.643119 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" event={"ID":"0cbd6e3c-fca8-40bb-a340-9b6433c0b077","Type":"ContainerStarted","Data":"c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8"} Jul 07 13:56:44.718632 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:44.718604 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:44.718848 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:44.718827 2576 patch_prober.go:28] interesting pod/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 07 13:56:44.718899 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:44.718875 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 07 13:56:45.719037 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:45.719005 2576 patch_prober.go:28] interesting pod/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 07 13:56:45.719443 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:45.719058 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 07 13:56:46.719506 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:46.719474 2576 patch_prober.go:28] interesting pod/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 07 13:56:46.719894 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:46.719527 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 07 13:56:47.718852 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:47.718817 2576 patch_prober.go:28] interesting pod/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 07 13:56:47.719029 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:47.718876 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 07 13:56:48.719346 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:48.719263 2576 patch_prober.go:28] interesting pod/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 07 13:56:48.719346 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:48.719322 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 07 13:56:49.722318 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:49.722287 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:49.722755 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:49.722566 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:49.723253 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:49.723235 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:50.018642 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:50.018610 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m"] Jul 07 13:56:50.679229 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:50.679200 2576 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" secret="" err="secret \"e2e-mt-0e8fd7f6-openshift-default-dockercfg-27vfr\" not found" Jul 07 13:56:51.684035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:51.683989 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" containerID="cri-o://c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8" gracePeriod=30 Jul 07 13:56:56.948412 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:56.948390 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:57.140266 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140236 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-certs\") pod \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " Jul 07 13:56:57.140266 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140273 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-data\") pod \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " Jul 07 13:56:57.140483 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140310 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-envoy\") pod \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " Jul 07 13:56:57.140483 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140349 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-token\") pod \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " Jul 07 13:56:57.140483 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140372 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istiod-ca-cert\") pod \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " Jul 07 13:56:57.140483 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140392 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ptlj\" (UniqueName: \"kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-kube-api-access-9ptlj\") pod \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " Jul 07 13:56:57.140483 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140454 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-credential-socket\") pod \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " Jul 07 13:56:57.140741 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140500 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-podinfo\") pod \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " Jul 07 13:56:57.140741 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140527 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-socket\") pod \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\" (UID: \"0cbd6e3c-fca8-40bb-a340-9b6433c0b077\") " Jul 07 13:56:57.140741 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140544 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0cbd6e3c-fca8-40bb-a340-9b6433c0b077" (UID: "0cbd6e3c-fca8-40bb-a340-9b6433c0b077"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:56:57.140896 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140770 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0cbd6e3c-fca8-40bb-a340-9b6433c0b077" (UID: "0cbd6e3c-fca8-40bb-a340-9b6433c0b077"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:56:57.140896 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140813 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:57.140896 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140807 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0cbd6e3c-fca8-40bb-a340-9b6433c0b077" (UID: "0cbd6e3c-fca8-40bb-a340-9b6433c0b077"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:56:57.141065 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.140995 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0cbd6e3c-fca8-40bb-a340-9b6433c0b077" (UID: "0cbd6e3c-fca8-40bb-a340-9b6433c0b077"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:56:57.141123 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.141063 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-data" (OuterVolumeSpecName: "istio-data") pod "0cbd6e3c-fca8-40bb-a340-9b6433c0b077" (UID: "0cbd6e3c-fca8-40bb-a340-9b6433c0b077"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:56:57.142717 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.142681 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-kube-api-access-9ptlj" (OuterVolumeSpecName: "kube-api-access-9ptlj") pod "0cbd6e3c-fca8-40bb-a340-9b6433c0b077" (UID: "0cbd6e3c-fca8-40bb-a340-9b6433c0b077"). InnerVolumeSpecName "kube-api-access-9ptlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:56:57.142717 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.142685 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0cbd6e3c-fca8-40bb-a340-9b6433c0b077" (UID: "0cbd6e3c-fca8-40bb-a340-9b6433c0b077"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 13:56:57.142896 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.142818 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-token" (OuterVolumeSpecName: "istio-token") pod "0cbd6e3c-fca8-40bb-a340-9b6433c0b077" (UID: "0cbd6e3c-fca8-40bb-a340-9b6433c0b077"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:56:57.142958 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.142940 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0cbd6e3c-fca8-40bb-a340-9b6433c0b077" (UID: "0cbd6e3c-fca8-40bb-a340-9b6433c0b077"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:56:57.241845 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.241819 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:57.241845 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.241841 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ptlj\" (UniqueName: \"kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-kube-api-access-9ptlj\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:57.242004 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.241852 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:57.242004 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.241861 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:57.242004 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.241869 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:57.242004 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.241877 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:57.242004 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.241885 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:57.242004 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.241892 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbd6e3c-fca8-40bb-a340-9b6433c0b077-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:56:57.714640 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.714605 2576 generic.go:356] "Generic (PLEG): container finished" podID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerID="c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8" exitCode=0 Jul 07 13:56:57.714819 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.714653 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" Jul 07 13:56:57.714819 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.714694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" event={"ID":"0cbd6e3c-fca8-40bb-a340-9b6433c0b077","Type":"ContainerDied","Data":"c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8"} Jul 07 13:56:57.714819 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.714748 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m" event={"ID":"0cbd6e3c-fca8-40bb-a340-9b6433c0b077","Type":"ContainerDied","Data":"ecef18f9fe6d9591c1d1e8e7b4a99d6faadcced1d5f6f8ac983a4aa128c5f991"} Jul 07 13:56:57.714819 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.714771 2576 scope.go:117] "RemoveContainer" containerID="c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8" Jul 07 13:56:57.723698 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.723678 2576 scope.go:117] "RemoveContainer" containerID="3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0" Jul 07 13:56:57.731392 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.731378 2576 scope.go:117] "RemoveContainer" containerID="c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8" Jul 07 13:56:57.731637 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:56:57.731619 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8\": container with ID starting with c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8 not found: ID does not exist" containerID="c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8" Jul 07 13:56:57.731681 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.731647 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8"} err="failed to get container status \"c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8\": rpc error: code = NotFound desc = could not find container \"c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8\": container with ID starting with c87aec015dc36fe60de28477a6d78179c4c7cb3268fe5703520b34e483fa0cc8 not found: ID does not exist" Jul 07 13:56:57.731681 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.731664 2576 scope.go:117] "RemoveContainer" containerID="3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0" Jul 07 13:56:57.731900 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:56:57.731884 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0\": container with ID starting with 3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0 not found: ID does not exist" containerID="3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0" Jul 07 13:56:57.731938 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.731907 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0"} err="failed to get container status \"3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0\": rpc error: code = NotFound desc = could not find container \"3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0\": container with ID starting with 3df6e3188cb3e23bde6ea751c1effdd4e1359992f0aee06629f8aa59b45b11f0 not found: ID does not exist" Jul 07 13:56:57.747521 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.747501 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m"] Jul 07 13:56:57.749789 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:57.749770 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0e8fd7f6-openshift-default-745646f8c9-25l4m"] Jul 07 13:56:58.593526 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:56:58.593497 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" path="/var/lib/kubelet/pods/0cbd6e3c-fca8-40bb-a340-9b6433c0b077/volumes" Jul 07 13:57:11.436428 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.436394 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4"] Jul 07 13:57:11.437003 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.436982 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" Jul 07 13:57:11.437066 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.437007 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" Jul 07 13:57:11.437066 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.437017 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="120136d0-36d6-45fe-b1a6-4dc551c6531c" containerName="maas-api" Jul 07 13:57:11.437066 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.437023 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="120136d0-36d6-45fe-b1a6-4dc551c6531c" containerName="maas-api" Jul 07 13:57:11.437066 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.437043 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" Jul 07 13:57:11.437066 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.437050 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" Jul 07 13:57:11.437340 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.437112 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" Jul 07 13:57:11.437340 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.437120 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cbd6e3c-fca8-40bb-a340-9b6433c0b077" containerName="istio-proxy" Jul 07 13:57:11.437340 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.437127 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="120136d0-36d6-45fe-b1a6-4dc551c6531c" containerName="maas-api" Jul 07 13:57:11.441181 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.441150 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.444669 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.444651 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-52dd7e06-openshift-default-dockercfg-6d9zj\"" Jul 07 13:57:11.454637 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.454610 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4"] Jul 07 13:57:11.458935 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.458908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-credential-socket\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.459051 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.458938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-data\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.459051 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.458968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d890159-572f-4875-b004-34f7074e47ce-istio-podinfo\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.459195 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.459045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-certs\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.459195 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.459102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-istio-token\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.459195 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.459130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4k5b\" (UniqueName: \"kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-kube-api-access-m4k5b\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.459195 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.459179 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-socket\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.459427 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.459202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-envoy\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.459427 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.459223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d890159-572f-4875-b004-34f7074e47ce-istiod-ca-cert\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.559943 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.559908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-certs\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560100 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.559969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-istio-token\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560100 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4k5b\" (UniqueName: \"kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-kube-api-access-m4k5b\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560100 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-socket\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560100 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-envoy\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560343 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d890159-572f-4875-b004-34f7074e47ce-istiod-ca-cert\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560343 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-credential-socket\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560343 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-data\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560498 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560453 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-socket\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560576 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560553 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-data\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560632 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d890159-572f-4875-b004-34f7074e47ce-istio-podinfo\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560689 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560668 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-certs\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560689 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560674 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-credential-socket\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.560840 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.560822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d890159-572f-4875-b004-34f7074e47ce-istiod-ca-cert\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.562529 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.562512 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-envoy\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.562628 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.562612 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d890159-572f-4875-b004-34f7074e47ce-istio-podinfo\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.570795 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.570773 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4k5b\" (UniqueName: \"kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-kube-api-access-m4k5b\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.573252 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.573230 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-istio-token\") pod \"e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.751685 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.751654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:11.899361 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.899328 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4"] Jul 07 13:57:11.901802 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:57:11.901776 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d890159_572f_4875_b004_34f7074e47ce.slice/crio-3eeab87a0712810bdf3107fe29bf5cd8ebcb405bf32d507b0017f3cd58de5541 WatchSource:0}: Error finding container 3eeab87a0712810bdf3107fe29bf5cd8ebcb405bf32d507b0017f3cd58de5541: Status 404 returned error can't find the container with id 3eeab87a0712810bdf3107fe29bf5cd8ebcb405bf32d507b0017f3cd58de5541 Jul 07 13:57:11.903820 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.903790 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:57:11.903896 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.903874 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:57:11.903938 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:11.903902 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:57:12.788525 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:12.788494 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" event={"ID":"7d890159-572f-4875-b004-34f7074e47ce","Type":"ContainerStarted","Data":"ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1"} Jul 07 13:57:12.788984 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:12.788555 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" event={"ID":"7d890159-572f-4875-b004-34f7074e47ce","Type":"ContainerStarted","Data":"3eeab87a0712810bdf3107fe29bf5cd8ebcb405bf32d507b0017f3cd58de5541"} Jul 07 13:57:12.815041 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:12.814994 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" podStartSLOduration=1.814979898 podStartE2EDuration="1.814979898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:57:12.812985987 +0000 UTC m=+1766.814227515" watchObservedRunningTime="2026-07-07 13:57:12.814979898 +0000 UTC m=+1766.816221479" Jul 07 13:57:13.752544 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:13.752506 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:13.757509 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:13.757487 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:13.793218 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:13.793193 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:13.793970 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:13.793952 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:14.900761 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:14.900724 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq"] Jul 07 13:57:14.903861 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:14.903840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:14.907026 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:14.907009 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-52dd7e06-serving-cert\"" Jul 07 13:57:14.917631 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:14.917611 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq"] Jul 07 13:57:14.993698 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:14.993671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d21682-931c-4829-996a-be89791de8b0-maas-api-tls\") pod \"maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:14.993851 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:14.993721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d21682-931c-4829-996a-be89791de8b0-openshift-service-ca\") pod \"maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:14.993851 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:14.993784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwqz\" (UniqueName: \"kubernetes.io/projected/a4d21682-931c-4829-996a-be89791de8b0-kube-api-access-7lwqz\") pod \"maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:15.094770 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:15.094736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d21682-931c-4829-996a-be89791de8b0-maas-api-tls\") pod \"maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:15.094939 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:15.094786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d21682-931c-4829-996a-be89791de8b0-openshift-service-ca\") pod \"maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:15.094939 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:15.094815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwqz\" (UniqueName: \"kubernetes.io/projected/a4d21682-931c-4829-996a-be89791de8b0-kube-api-access-7lwqz\") pod \"maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:15.094939 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:57:15.094917 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-52dd7e06-serving-cert: secret "maas-api-e2e-mt-52dd7e06-serving-cert" not found Jul 07 13:57:15.095057 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:57:15.094984 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4d21682-931c-4829-996a-be89791de8b0-maas-api-tls podName:a4d21682-931c-4829-996a-be89791de8b0 nodeName:}" failed. No retries permitted until 2026-07-07 13:57:15.594965123 +0000 UTC m=+1769.596206635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4d21682-931c-4829-996a-be89791de8b0-maas-api-tls") pod "maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" (UID: "a4d21682-931c-4829-996a-be89791de8b0") : secret "maas-api-e2e-mt-52dd7e06-serving-cert" not found Jul 07 13:57:15.095494 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:15.095472 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d21682-931c-4829-996a-be89791de8b0-openshift-service-ca\") pod \"maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:15.104847 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:15.104822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwqz\" (UniqueName: \"kubernetes.io/projected/a4d21682-931c-4829-996a-be89791de8b0-kube-api-access-7lwqz\") pod \"maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:15.598662 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:15.598629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d21682-931c-4829-996a-be89791de8b0-maas-api-tls\") pod \"maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:15.601235 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:15.601201 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d21682-931c-4829-996a-be89791de8b0-maas-api-tls\") pod \"maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:15.815287 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:15.815256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:15.953829 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:15.953806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq"] Jul 07 13:57:15.955758 ip-10-0-132-236 kubenswrapper[2576]: W0707 13:57:15.955719 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d21682_931c_4829_996a_be89791de8b0.slice/crio-78e67a3a0f9424dbb58c15a461dc6c34c3c2092ac11227715b1a3a7938afc321 WatchSource:0}: Error finding container 78e67a3a0f9424dbb58c15a461dc6c34c3c2092ac11227715b1a3a7938afc321: Status 404 returned error can't find the container with id 78e67a3a0f9424dbb58c15a461dc6c34c3c2092ac11227715b1a3a7938afc321 Jul 07 13:57:16.810794 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:16.810760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" event={"ID":"a4d21682-931c-4829-996a-be89791de8b0","Type":"ContainerStarted","Data":"8d1c414abbd8d7abdee8417eb9ec8e23b79975a53edde71c45fef0c71cedff8e"} Jul 07 13:57:16.810960 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:16.810814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" event={"ID":"a4d21682-931c-4829-996a-be89791de8b0","Type":"ContainerStarted","Data":"78e67a3a0f9424dbb58c15a461dc6c34c3c2092ac11227715b1a3a7938afc321"} Jul 07 13:57:16.810960 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:16.810835 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:16.837073 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:16.837029 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" podStartSLOduration=2.465486341 podStartE2EDuration="2.83701375s" totalImagesPullingTime="371.527409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:57:16.83633225 +0000 UTC m=+1770.837573778" watchObservedRunningTime="2026-07-07 13:57:16.83701375 +0000 UTC m=+1770.838255277" Jul 07 13:57:22.821806 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:22.821775 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:57:28.795382 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:28.795351 2576 patch_prober.go:28] interesting pod/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 07 13:57:28.795696 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:28.795414 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 07 13:57:28.876500 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:28.876447 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d890159-572f-4875-b004-34f7074e47ce" containerID="ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1" exitCode=0 Jul 07 13:57:28.876577 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:28.876508 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" event={"ID":"7d890159-572f-4875-b004-34f7074e47ce","Type":"ContainerDied","Data":"ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1"} Jul 07 13:57:28.876877 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:28.876863 2576 scope.go:117] "RemoveContainer" containerID="ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1" Jul 07 13:57:28.877850 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:28.877819 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:57:28.877908 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:28.877897 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:57:28.877946 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:28.877932 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 13:57:29.884294 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:29.884263 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" event={"ID":"7d890159-572f-4875-b004-34f7074e47ce","Type":"ContainerStarted","Data":"ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e"} Jul 07 13:57:30.752845 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:30.752760 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:30.752845 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:30.752801 2576 patch_prober.go:28] interesting pod/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 07 13:57:30.753110 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:30.752858 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 07 13:57:31.752531 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:31.752495 2576 patch_prober.go:28] interesting pod/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 07 13:57:31.753093 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:31.753062 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 07 13:57:32.752875 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:32.752836 2576 patch_prober.go:28] interesting pod/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 07 13:57:32.753479 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:32.753446 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 07 13:57:33.755846 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:33.755814 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:33.756359 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:33.756217 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:57:33.756843 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:57:33.756825 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:58:07.871899 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:07.871868 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq"] Jul 07 13:58:07.872454 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:07.872064 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" podUID="a4d21682-931c-4829-996a-be89791de8b0" containerName="maas-api" containerID="cri-o://8d1c414abbd8d7abdee8417eb9ec8e23b79975a53edde71c45fef0c71cedff8e" gracePeriod=30 Jul 07 13:58:08.072683 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.072651 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4d21682-931c-4829-996a-be89791de8b0" containerID="8d1c414abbd8d7abdee8417eb9ec8e23b79975a53edde71c45fef0c71cedff8e" exitCode=0 Jul 07 13:58:08.072825 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.072724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" event={"ID":"a4d21682-931c-4829-996a-be89791de8b0","Type":"ContainerDied","Data":"8d1c414abbd8d7abdee8417eb9ec8e23b79975a53edde71c45fef0c71cedff8e"} Jul 07 13:58:08.120233 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.120205 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:58:08.270883 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.270857 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d21682-931c-4829-996a-be89791de8b0-openshift-service-ca\") pod \"a4d21682-931c-4829-996a-be89791de8b0\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " Jul 07 13:58:08.271068 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.270924 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d21682-931c-4829-996a-be89791de8b0-maas-api-tls\") pod \"a4d21682-931c-4829-996a-be89791de8b0\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " Jul 07 13:58:08.271068 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.270988 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lwqz\" (UniqueName: \"kubernetes.io/projected/a4d21682-931c-4829-996a-be89791de8b0-kube-api-access-7lwqz\") pod \"a4d21682-931c-4829-996a-be89791de8b0\" (UID: \"a4d21682-931c-4829-996a-be89791de8b0\") " Jul 07 13:58:08.271304 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.271272 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d21682-931c-4829-996a-be89791de8b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4d21682-931c-4829-996a-be89791de8b0" (UID: "a4d21682-931c-4829-996a-be89791de8b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:58:08.273063 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.273038 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d21682-931c-4829-996a-be89791de8b0-kube-api-access-7lwqz" (OuterVolumeSpecName: "kube-api-access-7lwqz") pod "a4d21682-931c-4829-996a-be89791de8b0" (UID: "a4d21682-931c-4829-996a-be89791de8b0"). InnerVolumeSpecName "kube-api-access-7lwqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:58:08.273152 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.273057 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d21682-931c-4829-996a-be89791de8b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4d21682-931c-4829-996a-be89791de8b0" (UID: "a4d21682-931c-4829-996a-be89791de8b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 13:58:08.371919 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.371885 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lwqz\" (UniqueName: \"kubernetes.io/projected/a4d21682-931c-4829-996a-be89791de8b0-kube-api-access-7lwqz\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:58:08.371919 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.371914 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d21682-931c-4829-996a-be89791de8b0-openshift-service-ca\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:58:08.371919 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:08.371925 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d21682-931c-4829-996a-be89791de8b0-maas-api-tls\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:58:09.078677 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:09.078636 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" event={"ID":"a4d21682-931c-4829-996a-be89791de8b0","Type":"ContainerDied","Data":"78e67a3a0f9424dbb58c15a461dc6c34c3c2092ac11227715b1a3a7938afc321"} Jul 07 13:58:09.079051 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:09.078726 2576 scope.go:117] "RemoveContainer" containerID="8d1c414abbd8d7abdee8417eb9ec8e23b79975a53edde71c45fef0c71cedff8e" Jul 07 13:58:09.079051 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:09.078657 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq" Jul 07 13:58:09.098175 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:09.098142 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq"] Jul 07 13:58:09.102757 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:09.102716 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-52dd7e06-6bf8fd4574-7nblq"] Jul 07 13:58:10.590997 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:58:10.590958 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d21682-931c-4829-996a-be89791de8b0" path="/var/lib/kubelet/pods/a4d21682-931c-4829-996a-be89791de8b0/volumes" Jul 07 13:59:03.319198 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:03.319148 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4"] Jul 07 13:59:03.319699 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:03.319426 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" containerID="cri-o://ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e" gracePeriod=30 Jul 07 13:59:03.758476 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:03.758439 2576 patch_prober.go:28] interesting pod/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 13:59:03.758641 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:03.758503 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 13:59:08.575110 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.575085 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:59:08.714334 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714245 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-certs\") pod \"7d890159-572f-4875-b004-34f7074e47ce\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " Jul 07 13:59:08.714334 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714300 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4k5b\" (UniqueName: \"kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-kube-api-access-m4k5b\") pod \"7d890159-572f-4875-b004-34f7074e47ce\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " Jul 07 13:59:08.714561 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714346 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-envoy\") pod \"7d890159-572f-4875-b004-34f7074e47ce\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " Jul 07 13:59:08.714561 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714373 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d890159-572f-4875-b004-34f7074e47ce-istio-podinfo\") pod \"7d890159-572f-4875-b004-34f7074e47ce\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " Jul 07 13:59:08.714561 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714405 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-istio-token\") pod \"7d890159-572f-4875-b004-34f7074e47ce\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " Jul 07 13:59:08.714561 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714460 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d890159-572f-4875-b004-34f7074e47ce-istiod-ca-cert\") pod \"7d890159-572f-4875-b004-34f7074e47ce\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " Jul 07 13:59:08.714561 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714500 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-data\") pod \"7d890159-572f-4875-b004-34f7074e47ce\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " Jul 07 13:59:08.714561 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714521 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d890159-572f-4875-b004-34f7074e47ce" (UID: "7d890159-572f-4875-b004-34f7074e47ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:59:08.714561 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714535 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-credential-socket\") pod \"7d890159-572f-4875-b004-34f7074e47ce\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " Jul 07 13:59:08.714920 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714597 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-socket\") pod \"7d890159-572f-4875-b004-34f7074e47ce\" (UID: \"7d890159-572f-4875-b004-34f7074e47ce\") " Jul 07 13:59:08.714998 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714979 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:59:08.715066 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.715048 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d890159-572f-4875-b004-34f7074e47ce" (UID: "7d890159-572f-4875-b004-34f7074e47ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:59:08.715132 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.714969 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d890159-572f-4875-b004-34f7074e47ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d890159-572f-4875-b004-34f7074e47ce" (UID: "7d890159-572f-4875-b004-34f7074e47ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 13:59:08.715210 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.715176 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d890159-572f-4875-b004-34f7074e47ce" (UID: "7d890159-572f-4875-b004-34f7074e47ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:59:08.715409 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.715383 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d890159-572f-4875-b004-34f7074e47ce" (UID: "7d890159-572f-4875-b004-34f7074e47ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:59:08.716655 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.716630 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d890159-572f-4875-b004-34f7074e47ce" (UID: "7d890159-572f-4875-b004-34f7074e47ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 13:59:08.716821 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.716800 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d890159-572f-4875-b004-34f7074e47ce" (UID: "7d890159-572f-4875-b004-34f7074e47ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:59:08.717028 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.716991 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d890159-572f-4875-b004-34f7074e47ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d890159-572f-4875-b004-34f7074e47ce" (UID: "7d890159-572f-4875-b004-34f7074e47ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 13:59:08.717028 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.716997 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-kube-api-access-m4k5b" (OuterVolumeSpecName: "kube-api-access-m4k5b") pod "7d890159-572f-4875-b004-34f7074e47ce" (UID: "7d890159-572f-4875-b004-34f7074e47ce"). InnerVolumeSpecName "kube-api-access-m4k5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 13:59:08.816366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.816335 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d890159-572f-4875-b004-34f7074e47ce-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:59:08.816366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.816363 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:59:08.816366 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.816373 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d890159-572f-4875-b004-34f7074e47ce-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:59:08.816593 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.816381 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:59:08.816593 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.816390 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:59:08.816593 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.816398 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:59:08.816593 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.816406 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4k5b\" (UniqueName: \"kubernetes.io/projected/7d890159-572f-4875-b004-34f7074e47ce-kube-api-access-m4k5b\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:59:08.816593 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:08.816414 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d890159-572f-4875-b004-34f7074e47ce-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 13:59:09.373035 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.373001 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d890159-572f-4875-b004-34f7074e47ce" containerID="ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e" exitCode=0 Jul 07 13:59:09.373243 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.373052 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" Jul 07 13:59:09.373243 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.373087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" event={"ID":"7d890159-572f-4875-b004-34f7074e47ce","Type":"ContainerDied","Data":"ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e"} Jul 07 13:59:09.373243 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.373139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4" event={"ID":"7d890159-572f-4875-b004-34f7074e47ce","Type":"ContainerDied","Data":"3eeab87a0712810bdf3107fe29bf5cd8ebcb405bf32d507b0017f3cd58de5541"} Jul 07 13:59:09.373243 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.373162 2576 scope.go:117] "RemoveContainer" containerID="ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e" Jul 07 13:59:09.382603 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.382583 2576 scope.go:117] "RemoveContainer" containerID="ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1" Jul 07 13:59:09.390811 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.390796 2576 scope.go:117] "RemoveContainer" containerID="ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e" Jul 07 13:59:09.391042 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:59:09.391025 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e\": container with ID starting with ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e not found: ID does not exist" containerID="ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e" Jul 07 13:59:09.391107 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.391048 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e"} err="failed to get container status \"ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e\": rpc error: code = NotFound desc = could not find container \"ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e\": container with ID starting with ce695be572ea217091ee567adb4241b862709c14c9b72b6e8e64cfdb191e808e not found: ID does not exist" Jul 07 13:59:09.391107 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.391065 2576 scope.go:117] "RemoveContainer" containerID="ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1" Jul 07 13:59:09.391330 ip-10-0-132-236 kubenswrapper[2576]: E0707 13:59:09.391311 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1\": container with ID starting with ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1 not found: ID does not exist" containerID="ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1" Jul 07 13:59:09.391371 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.391338 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1"} err="failed to get container status \"ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1\": rpc error: code = NotFound desc = could not find container \"ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1\": container with ID starting with ac82486771504f56ffc5395a34bf87bc36975103d950d543011c6eea4b49efd1 not found: ID does not exist" Jul 07 13:59:09.397530 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.397506 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4"] Jul 07 13:59:09.409585 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:09.409564 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-52dd7e06-openshift-default-99dfb69dc-qcmv4"] Jul 07 13:59:10.589872 ip-10-0-132-236 kubenswrapper[2576]: I0707 13:59:10.589837 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d890159-572f-4875-b004-34f7074e47ce" path="/var/lib/kubelet/pods/7d890159-572f-4875-b004-34f7074e47ce/volumes" Jul 07 14:00:00.152206 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.152159 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29723880-czfzp"] Jul 07 14:00:00.152969 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.152549 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d21682-931c-4829-996a-be89791de8b0" containerName="maas-api" Jul 07 14:00:00.152969 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.152560 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d21682-931c-4829-996a-be89791de8b0" containerName="maas-api" Jul 07 14:00:00.152969 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.152567 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" Jul 07 14:00:00.152969 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.152575 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" Jul 07 14:00:00.152969 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.152645 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d21682-931c-4829-996a-be89791de8b0" containerName="maas-api" Jul 07 14:00:00.152969 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.152654 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" Jul 07 14:00:00.152969 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.152661 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" Jul 07 14:00:00.155020 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.154999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" Jul 07 14:00:00.166244 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.166220 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723880-czfzp"] Jul 07 14:00:00.249321 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.249292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xwm\" (UniqueName: \"kubernetes.io/projected/e2bef877-d26f-433b-a68a-cf29ae46737a-kube-api-access-s8xwm\") pod \"maas-api-key-cleanup-29723880-czfzp\" (UID: \"e2bef877-d26f-433b-a68a-cf29ae46737a\") " pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" Jul 07 14:00:00.350089 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.350055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xwm\" (UniqueName: \"kubernetes.io/projected/e2bef877-d26f-433b-a68a-cf29ae46737a-kube-api-access-s8xwm\") pod \"maas-api-key-cleanup-29723880-czfzp\" (UID: \"e2bef877-d26f-433b-a68a-cf29ae46737a\") " pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" Jul 07 14:00:00.363243 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.363217 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xwm\" (UniqueName: \"kubernetes.io/projected/e2bef877-d26f-433b-a68a-cf29ae46737a-kube-api-access-s8xwm\") pod \"maas-api-key-cleanup-29723880-czfzp\" (UID: \"e2bef877-d26f-433b-a68a-cf29ae46737a\") " pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" Jul 07 14:00:00.412904 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.412835 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm"] Jul 07 14:00:00.413228 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.413215 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" Jul 07 14:00:00.413277 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.413229 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d890159-572f-4875-b004-34f7074e47ce" containerName="istio-proxy" Jul 07 14:00:00.415897 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.415879 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.419851 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.419832 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-102141-openshift-default-dockercfg-bmttx\"" Jul 07 14:00:00.435048 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.435021 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm"] Jul 07 14:00:00.464689 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.464666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" Jul 07 14:00:00.552761 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.552730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-credential-socket\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.552938 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.552765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhtnn\" (UniqueName: \"kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-kube-api-access-zhtnn\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.552938 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.552884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-certs\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.553058 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.552936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-socket\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.553058 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.552963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-podinfo\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.553058 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.552987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-token\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.553058 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.553037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-data\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.553297 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.553080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istiod-ca-cert\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.553297 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.553101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-envoy\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.654355 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-envoy\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.654521 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-credential-socket\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.654521 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhtnn\" (UniqueName: \"kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-kube-api-access-zhtnn\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.654605 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-certs\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.654605 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-socket\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.654707 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-podinfo\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.654707 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-token\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.654829 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-data\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.654885 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istiod-ca-cert\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.654964 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654901 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-certs\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.655025 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.654958 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-socket\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.655233 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.655208 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-credential-socket\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.655341 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.655226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-data\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.655495 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.655477 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istiod-ca-cert\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.656685 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.656657 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-envoy\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.657072 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.657053 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-podinfo\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.664180 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.664088 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-token\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.664284 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.664189 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhtnn\" (UniqueName: \"kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-kube-api-access-zhtnn\") pod \"e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.726470 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.726445 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:00.815797 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.815770 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723880-czfzp"] Jul 07 14:00:00.818279 ip-10-0-132-236 kubenswrapper[2576]: W0707 14:00:00.818234 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bef877_d26f_433b_a68a_cf29ae46737a.slice/crio-c810064d8da3f2eb7435559d2e4e5cb6f9ea7e4a6800a8374588084a90de0734 WatchSource:0}: Error finding container c810064d8da3f2eb7435559d2e4e5cb6f9ea7e4a6800a8374588084a90de0734: Status 404 returned error can't find the container with id c810064d8da3f2eb7435559d2e4e5cb6f9ea7e4a6800a8374588084a90de0734 Jul 07 14:00:00.872024 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.871994 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm"] Jul 07 14:00:00.873242 ip-10-0-132-236 kubenswrapper[2576]: W0707 14:00:00.873217 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1c722a_a9ac_4ad1_83e9_aa099fa26488.slice/crio-e034c96ad668c8326dd139d6b42802fb41c2f6fda1883e31b8a99651552118bd WatchSource:0}: Error finding container e034c96ad668c8326dd139d6b42802fb41c2f6fda1883e31b8a99651552118bd: Status 404 returned error can't find the container with id e034c96ad668c8326dd139d6b42802fb41c2f6fda1883e31b8a99651552118bd Jul 07 14:00:00.875409 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.875373 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:00.875508 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.875441 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:00.875508 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:00.875469 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:01.620921 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:01.620888 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" event={"ID":"e2bef877-d26f-433b-a68a-cf29ae46737a","Type":"ContainerStarted","Data":"13aa1f7b61c35274a0d3bc949d6fc47acdb47b17509dfcb82869a4c51fcfdfb1"} Jul 07 14:00:01.621433 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:01.620947 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" event={"ID":"e2bef877-d26f-433b-a68a-cf29ae46737a","Type":"ContainerStarted","Data":"c810064d8da3f2eb7435559d2e4e5cb6f9ea7e4a6800a8374588084a90de0734"} Jul 07 14:00:01.622563 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:01.622531 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" event={"ID":"cf1c722a-a9ac-4ad1-83e9-aa099fa26488","Type":"ContainerStarted","Data":"bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111"} Jul 07 14:00:01.622683 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:01.622590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" event={"ID":"cf1c722a-a9ac-4ad1-83e9-aa099fa26488","Type":"ContainerStarted","Data":"e034c96ad668c8326dd139d6b42802fb41c2f6fda1883e31b8a99651552118bd"} Jul 07 14:00:01.642685 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:01.642638 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" podStartSLOduration=1.642622624 podStartE2EDuration="1.642622624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:00:01.640346262 +0000 UTC m=+1935.641587794" watchObservedRunningTime="2026-07-07 14:00:01.642622624 +0000 UTC m=+1935.643864151" Jul 07 14:00:01.667071 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:01.667014 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" podStartSLOduration=1.6669939440000001 podStartE2EDuration="1.666993944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:00:01.665467557 +0000 UTC m=+1935.666709084" watchObservedRunningTime="2026-07-07 14:00:01.666993944 +0000 UTC m=+1935.668235473" Jul 07 14:00:01.727249 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:01.727197 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:01.728921 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:01.728892 2576 patch_prober.go:28] interesting pod/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 07 14:00:01.729077 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:01.728946 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 07 14:00:02.628653 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:02.628615 2576 generic.go:356] "Generic (PLEG): container finished" podID="e2bef877-d26f-433b-a68a-cf29ae46737a" containerID="13aa1f7b61c35274a0d3bc949d6fc47acdb47b17509dfcb82869a4c51fcfdfb1" exitCode=0 Jul 07 14:00:02.629037 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:02.628689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" event={"ID":"e2bef877-d26f-433b-a68a-cf29ae46737a","Type":"ContainerDied","Data":"13aa1f7b61c35274a0d3bc949d6fc47acdb47b17509dfcb82869a4c51fcfdfb1"} Jul 07 14:00:02.730750 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:02.730720 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:03.634254 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.634222 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:03.635309 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.635279 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:03.792921 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.792894 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" Jul 07 14:00:03.831480 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.831448 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh"] Jul 07 14:00:03.832264 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.832239 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2bef877-d26f-433b-a68a-cf29ae46737a" containerName="cleanup" Jul 07 14:00:03.832264 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.832262 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2bef877-d26f-433b-a68a-cf29ae46737a" containerName="cleanup" Jul 07 14:00:03.832453 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.832366 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2bef877-d26f-433b-a68a-cf29ae46737a" containerName="cleanup" Jul 07 14:00:03.835051 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.835029 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:03.838485 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.838460 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-102141-serving-cert\"" Jul 07 14:00:03.848601 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.848574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh"] Jul 07 14:00:03.886217 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.886111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0917e8a-58b2-40f0-94ca-3daf81a4affa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-102141-9f7469574-46ljh\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:03.886217 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.886149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0917e8a-58b2-40f0-94ca-3daf81a4affa-maas-api-tls\") pod \"maas-api-e2e-inf-a-102141-9f7469574-46ljh\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:03.886415 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.886255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxfx9\" (UniqueName: \"kubernetes.io/projected/b0917e8a-58b2-40f0-94ca-3daf81a4affa-kube-api-access-dxfx9\") pod \"maas-api-e2e-inf-a-102141-9f7469574-46ljh\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:03.986995 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.986954 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8xwm\" (UniqueName: \"kubernetes.io/projected/e2bef877-d26f-433b-a68a-cf29ae46737a-kube-api-access-s8xwm\") pod \"e2bef877-d26f-433b-a68a-cf29ae46737a\" (UID: \"e2bef877-d26f-433b-a68a-cf29ae46737a\") " Jul 07 14:00:03.987209 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.987058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0917e8a-58b2-40f0-94ca-3daf81a4affa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-102141-9f7469574-46ljh\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:03.987209 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.987080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0917e8a-58b2-40f0-94ca-3daf81a4affa-maas-api-tls\") pod \"maas-api-e2e-inf-a-102141-9f7469574-46ljh\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:03.987209 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.987110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxfx9\" (UniqueName: \"kubernetes.io/projected/b0917e8a-58b2-40f0-94ca-3daf81a4affa-kube-api-access-dxfx9\") pod \"maas-api-e2e-inf-a-102141-9f7469574-46ljh\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:03.987327 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:00:03.987263 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-102141-serving-cert: secret "maas-api-e2e-inf-a-102141-serving-cert" not found Jul 07 14:00:03.987360 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:00:03.987353 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0917e8a-58b2-40f0-94ca-3daf81a4affa-maas-api-tls podName:b0917e8a-58b2-40f0-94ca-3daf81a4affa nodeName:}" failed. No retries permitted until 2026-07-07 14:00:04.487332393 +0000 UTC m=+1938.488573898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0917e8a-58b2-40f0-94ca-3daf81a4affa-maas-api-tls") pod "maas-api-e2e-inf-a-102141-9f7469574-46ljh" (UID: "b0917e8a-58b2-40f0-94ca-3daf81a4affa") : secret "maas-api-e2e-inf-a-102141-serving-cert" not found Jul 07 14:00:03.987853 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.987829 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0917e8a-58b2-40f0-94ca-3daf81a4affa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-102141-9f7469574-46ljh\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:03.989138 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:03.989116 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bef877-d26f-433b-a68a-cf29ae46737a-kube-api-access-s8xwm" (OuterVolumeSpecName: "kube-api-access-s8xwm") pod "e2bef877-d26f-433b-a68a-cf29ae46737a" (UID: "e2bef877-d26f-433b-a68a-cf29ae46737a"). InnerVolumeSpecName "kube-api-access-s8xwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:00:04.002280 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:04.002258 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxfx9\" (UniqueName: \"kubernetes.io/projected/b0917e8a-58b2-40f0-94ca-3daf81a4affa-kube-api-access-dxfx9\") pod \"maas-api-e2e-inf-a-102141-9f7469574-46ljh\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:04.087733 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:04.087694 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8xwm\" (UniqueName: \"kubernetes.io/projected/e2bef877-d26f-433b-a68a-cf29ae46737a-kube-api-access-s8xwm\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:00:04.492384 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:04.492333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0917e8a-58b2-40f0-94ca-3daf81a4affa-maas-api-tls\") pod \"maas-api-e2e-inf-a-102141-9f7469574-46ljh\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:04.494929 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:04.494904 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0917e8a-58b2-40f0-94ca-3daf81a4affa-maas-api-tls\") pod \"maas-api-e2e-inf-a-102141-9f7469574-46ljh\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:04.640914 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:04.640874 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" Jul 07 14:00:04.641333 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:04.640932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723880-czfzp" event={"ID":"e2bef877-d26f-433b-a68a-cf29ae46737a","Type":"ContainerDied","Data":"c810064d8da3f2eb7435559d2e4e5cb6f9ea7e4a6800a8374588084a90de0734"} Jul 07 14:00:04.641333 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:04.640990 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c810064d8da3f2eb7435559d2e4e5cb6f9ea7e4a6800a8374588084a90de0734" Jul 07 14:00:04.745983 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:04.745888 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:04.896740 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:04.896660 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh"] Jul 07 14:00:04.899534 ip-10-0-132-236 kubenswrapper[2576]: W0707 14:00:04.899507 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0917e8a_58b2_40f0_94ca_3daf81a4affa.slice/crio-a615f8981b4d9288522147bccf366f6f157f41aa5c366efa1e43139e5cd53f25 WatchSource:0}: Error finding container a615f8981b4d9288522147bccf366f6f157f41aa5c366efa1e43139e5cd53f25: Status 404 returned error can't find the container with id a615f8981b4d9288522147bccf366f6f157f41aa5c366efa1e43139e5cd53f25 Jul 07 14:00:04.901029 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:04.900988 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:00:05.646640 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:05.646605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" event={"ID":"b0917e8a-58b2-40f0-94ca-3daf81a4affa","Type":"ContainerStarted","Data":"a615f8981b4d9288522147bccf366f6f157f41aa5c366efa1e43139e5cd53f25"} Jul 07 14:00:06.652879 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:06.652839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" event={"ID":"b0917e8a-58b2-40f0-94ca-3daf81a4affa","Type":"ContainerStarted","Data":"5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e"} Jul 07 14:00:06.653393 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:06.652975 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:06.675631 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:06.675575 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" podStartSLOduration=2.853160238 podStartE2EDuration="3.675554961s" totalImagesPullingTime="822.394723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:00:06.673381549 +0000 UTC m=+1940.674623074" watchObservedRunningTime="2026-07-07 14:00:06.675554961 +0000 UTC m=+1940.676796488" Jul 07 14:00:08.785947 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.785917 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562"] Jul 07 14:00:08.788542 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.788527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.791708 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.791690 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-da5db0-openshift-default-dockercfg-ghx4w\"" Jul 07 14:00:08.803610 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.803581 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562"] Jul 07 14:00:08.831937 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.831898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4a131c-5e02-4a28-934e-496d725a2c4a-istiod-ca-cert\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.831937 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.831941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-envoy\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.832217 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.831969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-token\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.832217 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.831999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q9jv\" (UniqueName: \"kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-kube-api-access-6q9jv\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.832217 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.832078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-certs\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.832217 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.832109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-credential-socket\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.837146 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.832143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-socket\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.837146 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.832604 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-podinfo\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.837146 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.833800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-data\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.934328 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.934290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4a131c-5e02-4a28-934e-496d725a2c4a-istiod-ca-cert\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.934328 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.934333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-envoy\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.934559 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.934354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-token\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.934559 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.934381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q9jv\" (UniqueName: \"kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-kube-api-access-6q9jv\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.934559 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.934450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-certs\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.934559 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.934485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-credential-socket\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.934559 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.934519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-socket\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.934559 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.934542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-podinfo\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.934885 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.934619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-data\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.935059 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.935029 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-credential-socket\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.935184 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.935036 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-certs\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.935184 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.935119 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-socket\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.935296 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.935192 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4a131c-5e02-4a28-934e-496d725a2c4a-istiod-ca-cert\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.935358 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.935341 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-data\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.936706 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.936680 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-envoy\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.936992 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.936974 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-podinfo\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.943989 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.943969 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-token\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:08.944320 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:08.944298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q9jv\" (UniqueName: \"kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-kube-api-access-6q9jv\") pod \"e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:09.098981 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:09.098900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:09.242218 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:09.242192 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562"] Jul 07 14:00:09.243981 ip-10-0-132-236 kubenswrapper[2576]: W0707 14:00:09.243939 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4a131c_5e02_4a28_934e_496d725a2c4a.slice/crio-a54d40e635a0b5dedef1c080aa0d2665aa457b29b7b24184f328d95fd9ec3498 WatchSource:0}: Error finding container a54d40e635a0b5dedef1c080aa0d2665aa457b29b7b24184f328d95fd9ec3498: Status 404 returned error can't find the container with id a54d40e635a0b5dedef1c080aa0d2665aa457b29b7b24184f328d95fd9ec3498 Jul 07 14:00:09.246287 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:09.246253 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:09.246367 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:09.246321 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:09.246367 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:09.246350 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:09.668521 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:09.668477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" event={"ID":"4a4a131c-5e02-4a28-934e-496d725a2c4a","Type":"ContainerStarted","Data":"eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665"} Jul 07 14:00:09.668721 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:09.668539 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" event={"ID":"4a4a131c-5e02-4a28-934e-496d725a2c4a","Type":"ContainerStarted","Data":"a54d40e635a0b5dedef1c080aa0d2665aa457b29b7b24184f328d95fd9ec3498"} Jul 07 14:00:09.693637 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:09.693589 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" podStartSLOduration=1.693572997 podStartE2EDuration="1.693572997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:00:09.691885829 +0000 UTC m=+1943.693127359" watchObservedRunningTime="2026-07-07 14:00:09.693572997 +0000 UTC m=+1943.694814663" Jul 07 14:00:10.099215 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:10.099182 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:10.104285 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:10.104256 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:10.673885 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:10.673859 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:10.674730 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:10.674712 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:12.664074 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:12.664045 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:00:19.677403 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:19.677351 2576 patch_prober.go:28] interesting pod/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 14:00:19.678005 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:19.677971 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 14:00:20.729517 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:20.729482 2576 generic.go:356] "Generic (PLEG): container finished" podID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerID="bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111" exitCode=0 Jul 07 14:00:20.729910 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:20.729559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" event={"ID":"cf1c722a-a9ac-4ad1-83e9-aa099fa26488","Type":"ContainerDied","Data":"bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111"} Jul 07 14:00:20.729956 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:20.729948 2576 scope.go:117] "RemoveContainer" containerID="bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111" Jul 07 14:00:20.730911 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:20.730885 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:20.730976 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:20.730951 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:20.731014 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:20.730983 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:21.736618 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:21.736583 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" event={"ID":"cf1c722a-a9ac-4ad1-83e9-aa099fa26488","Type":"ContainerStarted","Data":"9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778"} Jul 07 14:00:22.726674 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:22.726636 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:22.726879 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:22.726817 2576 patch_prober.go:28] interesting pod/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 07 14:00:22.726879 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:22.726857 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 07 14:00:23.727701 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:23.727665 2576 patch_prober.go:28] interesting pod/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 07 14:00:23.728089 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:23.727721 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 07 14:00:24.727330 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:24.727288 2576 patch_prober.go:28] interesting pod/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 07 14:00:24.727531 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:24.727344 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 07 14:00:25.727406 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:25.727365 2576 patch_prober.go:28] interesting pod/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 07 14:00:25.727789 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:25.727444 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 07 14:00:26.249049 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.249014 2576 patch_prober.go:28] interesting pod/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": read tcp 10.134.0.2:39488->10.134.0.62:15021: read: connection reset by peer" start-of-body= Jul 07 14:00:26.249220 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.249092 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": read tcp 10.134.0.2:39488->10.134.0.62:15021: read: connection reset by peer" Jul 07 14:00:26.765161 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.765130 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerID="eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665" exitCode=0 Jul 07 14:00:26.765556 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.765212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" event={"ID":"4a4a131c-5e02-4a28-934e-496d725a2c4a","Type":"ContainerDied","Data":"eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665"} Jul 07 14:00:26.765762 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.765746 2576 scope.go:117] "RemoveContainer" containerID="eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665" Jul 07 14:00:26.766691 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.766674 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:26.766822 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.766795 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:26.766893 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.766861 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:26.766954 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.766894 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:00:26.767009 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.766979 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:26.767620 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:26.767602 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:00:27.777210 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:27.777145 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" event={"ID":"4a4a131c-5e02-4a28-934e-496d725a2c4a","Type":"ContainerStarted","Data":"7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec"} Jul 07 14:00:28.100090 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:28.099996 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:28.100396 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:28.100372 2576 patch_prober.go:28] interesting pod/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 07 14:00:28.100455 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:28.100422 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 07 14:00:29.100230 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:29.100197 2576 patch_prober.go:28] interesting pod/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 07 14:00:29.100658 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:29.100261 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 07 14:00:30.099786 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:30.099749 2576 patch_prober.go:28] interesting pod/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 07 14:00:30.099963 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:30.099804 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 07 14:00:31.100268 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:31.100222 2576 patch_prober.go:28] interesting pod/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 07 14:00:31.100666 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:31.100290 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 07 14:00:32.103063 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:32.103032 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:32.103442 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:32.103354 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:00:32.104020 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:00:32.104006 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:01:22.312506 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.312475 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh"] Jul 07 14:01:22.312986 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.312675 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" podUID="b0917e8a-58b2-40f0-94ca-3daf81a4affa" containerName="maas-api" containerID="cri-o://5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e" gracePeriod=30 Jul 07 14:01:22.561352 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.561330 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:01:22.706878 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.706807 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0917e8a-58b2-40f0-94ca-3daf81a4affa-openshift-service-ca\") pod \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " Jul 07 14:01:22.707027 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.706924 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0917e8a-58b2-40f0-94ca-3daf81a4affa-maas-api-tls\") pod \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " Jul 07 14:01:22.707027 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.706976 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxfx9\" (UniqueName: \"kubernetes.io/projected/b0917e8a-58b2-40f0-94ca-3daf81a4affa-kube-api-access-dxfx9\") pod \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\" (UID: \"b0917e8a-58b2-40f0-94ca-3daf81a4affa\") " Jul 07 14:01:22.707191 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.707113 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0917e8a-58b2-40f0-94ca-3daf81a4affa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0917e8a-58b2-40f0-94ca-3daf81a4affa" (UID: "b0917e8a-58b2-40f0-94ca-3daf81a4affa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:01:22.707265 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.707216 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0917e8a-58b2-40f0-94ca-3daf81a4affa-openshift-service-ca\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:01:22.709041 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.709012 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0917e8a-58b2-40f0-94ca-3daf81a4affa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0917e8a-58b2-40f0-94ca-3daf81a4affa" (UID: "b0917e8a-58b2-40f0-94ca-3daf81a4affa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:01:22.709438 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.709418 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0917e8a-58b2-40f0-94ca-3daf81a4affa-kube-api-access-dxfx9" (OuterVolumeSpecName: "kube-api-access-dxfx9") pod "b0917e8a-58b2-40f0-94ca-3daf81a4affa" (UID: "b0917e8a-58b2-40f0-94ca-3daf81a4affa"). InnerVolumeSpecName "kube-api-access-dxfx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:01:22.808158 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.808130 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0917e8a-58b2-40f0-94ca-3daf81a4affa-maas-api-tls\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:01:22.808158 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:22.808156 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxfx9\" (UniqueName: \"kubernetes.io/projected/b0917e8a-58b2-40f0-94ca-3daf81a4affa-kube-api-access-dxfx9\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:01:23.051766 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:23.051729 2576 generic.go:356] "Generic (PLEG): container finished" podID="b0917e8a-58b2-40f0-94ca-3daf81a4affa" containerID="5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e" exitCode=0 Jul 07 14:01:23.051959 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:23.051772 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" Jul 07 14:01:23.051959 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:23.051822 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" event={"ID":"b0917e8a-58b2-40f0-94ca-3daf81a4affa","Type":"ContainerDied","Data":"5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e"} Jul 07 14:01:23.051959 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:23.051885 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh" event={"ID":"b0917e8a-58b2-40f0-94ca-3daf81a4affa","Type":"ContainerDied","Data":"a615f8981b4d9288522147bccf366f6f157f41aa5c366efa1e43139e5cd53f25"} Jul 07 14:01:23.051959 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:23.051917 2576 scope.go:117] "RemoveContainer" containerID="5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e" Jul 07 14:01:23.067098 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:23.067080 2576 scope.go:117] "RemoveContainer" containerID="5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e" Jul 07 14:01:23.067416 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:01:23.067381 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e\": container with ID starting with 5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e not found: ID does not exist" containerID="5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e" Jul 07 14:01:23.067482 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:23.067424 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e"} err="failed to get container status \"5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e\": rpc error: code = NotFound desc = could not find container \"5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e\": container with ID starting with 5adca74fe8fab6f1b0e471ebe54b2d58b90a9876372a13327e99504b8032a99e not found: ID does not exist" Jul 07 14:01:23.078877 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:23.078855 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh"] Jul 07 14:01:23.085808 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:23.085786 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-102141-9f7469574-46ljh"] Jul 07 14:01:24.590051 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:01:24.590018 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0917e8a-58b2-40f0-94ca-3daf81a4affa" path="/var/lib/kubelet/pods/b0917e8a-58b2-40f0-94ca-3daf81a4affa/volumes" Jul 07 14:02:04.920720 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:04.920689 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm"] Jul 07 14:02:04.921240 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:04.920903 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" containerID="cri-o://9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778" gracePeriod=30 Jul 07 14:02:10.188592 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.188523 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:02:10.308004 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.307966 2576 generic.go:356] "Generic (PLEG): container finished" podID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerID="9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778" exitCode=0 Jul 07 14:02:10.308197 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.308021 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" Jul 07 14:02:10.308197 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.308051 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" event={"ID":"cf1c722a-a9ac-4ad1-83e9-aa099fa26488","Type":"ContainerDied","Data":"9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778"} Jul 07 14:02:10.308197 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.308110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm" event={"ID":"cf1c722a-a9ac-4ad1-83e9-aa099fa26488","Type":"ContainerDied","Data":"e034c96ad668c8326dd139d6b42802fb41c2f6fda1883e31b8a99651552118bd"} Jul 07 14:02:10.308197 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.308134 2576 scope.go:117] "RemoveContainer" containerID="9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778" Jul 07 14:02:10.316763 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.316749 2576 scope.go:117] "RemoveContainer" containerID="bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111" Jul 07 14:02:10.324572 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.324557 2576 scope.go:117] "RemoveContainer" containerID="9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778" Jul 07 14:02:10.324815 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:02:10.324797 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778\": container with ID starting with 9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778 not found: ID does not exist" containerID="9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778" Jul 07 14:02:10.324861 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.324825 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778"} err="failed to get container status \"9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778\": rpc error: code = NotFound desc = could not find container \"9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778\": container with ID starting with 9f117ca970396283856752522438eec4352e0c9775fc000f423bd87a1f700778 not found: ID does not exist" Jul 07 14:02:10.324861 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.324840 2576 scope.go:117] "RemoveContainer" containerID="bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111" Jul 07 14:02:10.325046 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:02:10.325031 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111\": container with ID starting with bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111 not found: ID does not exist" containerID="bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111" Jul 07 14:02:10.325093 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.325050 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111"} err="failed to get container status \"bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111\": rpc error: code = NotFound desc = could not find container \"bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111\": container with ID starting with bf30714d8688d46c70ab572e04c4545c7ef3c61d180f47d0a3354d96ea037111 not found: ID does not exist" Jul 07 14:02:10.340926 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.340904 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-data\") pod \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " Jul 07 14:02:10.341010 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.340950 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-credential-socket\") pod \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " Jul 07 14:02:10.341010 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.340995 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-token\") pod \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " Jul 07 14:02:10.341086 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341067 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-podinfo\") pod \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " Jul 07 14:02:10.341120 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341103 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-certs\") pod \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " Jul 07 14:02:10.341156 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341132 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-socket\") pod \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " Jul 07 14:02:10.341216 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341199 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhtnn\" (UniqueName: \"kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-kube-api-access-zhtnn\") pod \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " Jul 07 14:02:10.341268 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341206 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf1c722a-a9ac-4ad1-83e9-aa099fa26488" (UID: "cf1c722a-a9ac-4ad1-83e9-aa099fa26488"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:10.341268 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341235 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-envoy\") pod \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " Jul 07 14:02:10.341374 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341285 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istiod-ca-cert\") pod \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\" (UID: \"cf1c722a-a9ac-4ad1-83e9-aa099fa26488\") " Jul 07 14:02:10.341429 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341379 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf1c722a-a9ac-4ad1-83e9-aa099fa26488" (UID: "cf1c722a-a9ac-4ad1-83e9-aa099fa26488"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:10.341485 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341458 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf1c722a-a9ac-4ad1-83e9-aa099fa26488" (UID: "cf1c722a-a9ac-4ad1-83e9-aa099fa26488"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:10.341680 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341655 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:10.341806 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341676 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf1c722a-a9ac-4ad1-83e9-aa099fa26488" (UID: "cf1c722a-a9ac-4ad1-83e9-aa099fa26488"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:02:10.341806 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341689 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:10.341806 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.341709 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:10.342078 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.342056 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf1c722a-a9ac-4ad1-83e9-aa099fa26488" (UID: "cf1c722a-a9ac-4ad1-83e9-aa099fa26488"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:10.343352 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.343328 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-kube-api-access-zhtnn" (OuterVolumeSpecName: "kube-api-access-zhtnn") pod "cf1c722a-a9ac-4ad1-83e9-aa099fa26488" (UID: "cf1c722a-a9ac-4ad1-83e9-aa099fa26488"). InnerVolumeSpecName "kube-api-access-zhtnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:02:10.343352 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.343320 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf1c722a-a9ac-4ad1-83e9-aa099fa26488" (UID: "cf1c722a-a9ac-4ad1-83e9-aa099fa26488"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:10.343548 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.343438 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf1c722a-a9ac-4ad1-83e9-aa099fa26488" (UID: "cf1c722a-a9ac-4ad1-83e9-aa099fa26488"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 14:02:10.343548 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.343522 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf1c722a-a9ac-4ad1-83e9-aa099fa26488" (UID: "cf1c722a-a9ac-4ad1-83e9-aa099fa26488"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:02:10.443110 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.443023 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:10.443110 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.443061 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:10.443110 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.443070 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:10.443110 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.443078 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:10.443110 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.443086 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhtnn\" (UniqueName: \"kubernetes.io/projected/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-kube-api-access-zhtnn\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:10.443110 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.443096 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf1c722a-a9ac-4ad1-83e9-aa099fa26488-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:10.636564 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.636529 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm"] Jul 07 14:02:10.638304 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:10.638284 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-102141-openshift-default-7db4b58f87-nqwwm"] Jul 07 14:02:12.590942 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:12.590909 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" path="/var/lib/kubelet/pods/cf1c722a-a9ac-4ad1-83e9-aa099fa26488/volumes" Jul 07 14:02:12.972497 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:12.972415 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562"] Jul 07 14:02:12.972764 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:12.972711 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" containerID="cri-o://7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec" gracePeriod=30 Jul 07 14:02:13.937673 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.937635 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x"] Jul 07 14:02:13.938061 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.938041 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" Jul 07 14:02:13.938061 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.938053 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" Jul 07 14:02:13.938061 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.938060 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" Jul 07 14:02:13.938205 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.938066 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" Jul 07 14:02:13.938205 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.938075 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0917e8a-58b2-40f0-94ca-3daf81a4affa" containerName="maas-api" Jul 07 14:02:13.938205 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.938080 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0917e8a-58b2-40f0-94ca-3daf81a4affa" containerName="maas-api" Jul 07 14:02:13.938205 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.938194 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" Jul 07 14:02:13.938205 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.938203 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0917e8a-58b2-40f0-94ca-3daf81a4affa" containerName="maas-api" Jul 07 14:02:13.940338 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.940320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:13.944384 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.944363 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d1d8a6-openshift-default-dockercfg-dz4z4\"" Jul 07 14:02:13.964390 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:13.964367 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x"] Jul 07 14:02:14.077120 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.077086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-socket\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.077120 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.077125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlj7d\" (UniqueName: \"kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-kube-api-access-jlj7d\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.077350 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.077146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58d742c-063d-4f71-a722-da67acc84a83-istiod-ca-cert\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.077350 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.077245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-certs\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.077350 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.077311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-credential-socket\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.077448 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.077356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-istio-token\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.077448 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.077398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58d742c-063d-4f71-a722-da67acc84a83-istio-podinfo\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.077515 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.077458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-data\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.077515 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.077479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-envoy\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.178491 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.178460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-data\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.178491 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.178495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-envoy\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.178693 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.178536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-socket\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.178693 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.178643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlj7d\" (UniqueName: \"kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-kube-api-access-jlj7d\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.178693 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.178687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58d742c-063d-4f71-a722-da67acc84a83-istiod-ca-cert\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.178805 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.178716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-certs\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.178805 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.178744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-credential-socket\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.178910 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.178812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-istio-token\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.178910 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.178878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58d742c-063d-4f71-a722-da67acc84a83-istio-podinfo\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.178910 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.178881 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-socket\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.179150 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.179123 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-credential-socket\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.179263 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.179141 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-certs\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.179444 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.179415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-data\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.179575 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.179454 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58d742c-063d-4f71-a722-da67acc84a83-istiod-ca-cert\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.180816 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.180798 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-envoy\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.181304 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.181287 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58d742c-063d-4f71-a722-da67acc84a83-istio-podinfo\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.190617 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.190558 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlj7d\" (UniqueName: \"kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-kube-api-access-jlj7d\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.192589 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.192565 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-istio-token\") pod \"e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.250473 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.250446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:14.395975 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.395948 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x"] Jul 07 14:02:14.397059 ip-10-0-132-236 kubenswrapper[2576]: W0707 14:02:14.397028 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58d742c_063d_4f71_a722_da67acc84a83.slice/crio-33a12eef6950a7cef8e8c1ae2a051ceb3f90fda85e8e6c01b98d7a239079bf1c WatchSource:0}: Error finding container 33a12eef6950a7cef8e8c1ae2a051ceb3f90fda85e8e6c01b98d7a239079bf1c: Status 404 returned error can't find the container with id 33a12eef6950a7cef8e8c1ae2a051ceb3f90fda85e8e6c01b98d7a239079bf1c Jul 07 14:02:14.399036 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.399003 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:14.399132 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.399093 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:14.399218 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:14.399140 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:15.336004 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:15.335970 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" event={"ID":"d58d742c-063d-4f71-a722-da67acc84a83","Type":"ContainerStarted","Data":"186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8"} Jul 07 14:02:15.336439 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:15.336020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" event={"ID":"d58d742c-063d-4f71-a722-da67acc84a83","Type":"ContainerStarted","Data":"33a12eef6950a7cef8e8c1ae2a051ceb3f90fda85e8e6c01b98d7a239079bf1c"} Jul 07 14:02:16.251262 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:16.251222 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:16.256281 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:16.256255 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:16.292658 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:16.292583 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" podStartSLOduration=3.292568332 podStartE2EDuration="3.292568332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:02:15.38198264 +0000 UTC m=+2069.383224165" watchObservedRunningTime="2026-07-07 14:02:16.292568332 +0000 UTC m=+2070.293809923" Jul 07 14:02:16.341092 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:16.341060 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:16.342140 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:16.342121 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:17.106395 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:17.106361 2576 patch_prober.go:28] interesting pod/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 14:02:17.106586 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:17.106425 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:02:18.221068 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.221045 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:02:18.317257 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317159 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-socket\") pod \"4a4a131c-5e02-4a28-934e-496d725a2c4a\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " Jul 07 14:02:18.317257 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317236 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4a131c-5e02-4a28-934e-496d725a2c4a-istiod-ca-cert\") pod \"4a4a131c-5e02-4a28-934e-496d725a2c4a\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " Jul 07 14:02:18.317461 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317285 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-token\") pod \"4a4a131c-5e02-4a28-934e-496d725a2c4a\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " Jul 07 14:02:18.317461 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317316 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-certs\") pod \"4a4a131c-5e02-4a28-934e-496d725a2c4a\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " Jul 07 14:02:18.317461 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317344 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-credential-socket\") pod \"4a4a131c-5e02-4a28-934e-496d725a2c4a\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " Jul 07 14:02:18.317461 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317383 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-envoy\") pod \"4a4a131c-5e02-4a28-934e-496d725a2c4a\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " Jul 07 14:02:18.317461 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317417 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-data\") pod \"4a4a131c-5e02-4a28-934e-496d725a2c4a\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " Jul 07 14:02:18.317720 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317484 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q9jv\" (UniqueName: \"kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-kube-api-access-6q9jv\") pod \"4a4a131c-5e02-4a28-934e-496d725a2c4a\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " Jul 07 14:02:18.317720 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317509 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-podinfo\") pod \"4a4a131c-5e02-4a28-934e-496d725a2c4a\" (UID: \"4a4a131c-5e02-4a28-934e-496d725a2c4a\") " Jul 07 14:02:18.317829 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317499 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a4a131c-5e02-4a28-934e-496d725a2c4a" (UID: "4a4a131c-5e02-4a28-934e-496d725a2c4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:18.317829 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317685 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a4a131c-5e02-4a28-934e-496d725a2c4a" (UID: "4a4a131c-5e02-4a28-934e-496d725a2c4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:18.317829 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317733 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4a131c-5e02-4a28-934e-496d725a2c4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a4a131c-5e02-4a28-934e-496d725a2c4a" (UID: "4a4a131c-5e02-4a28-934e-496d725a2c4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:02:18.317989 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.317916 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a4a131c-5e02-4a28-934e-496d725a2c4a" (UID: "4a4a131c-5e02-4a28-934e-496d725a2c4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:18.318272 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.318249 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a4a131c-5e02-4a28-934e-496d725a2c4a" (UID: "4a4a131c-5e02-4a28-934e-496d725a2c4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:18.319723 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.319698 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a4a131c-5e02-4a28-934e-496d725a2c4a" (UID: "4a4a131c-5e02-4a28-934e-496d725a2c4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:02:18.319808 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.319766 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a4a131c-5e02-4a28-934e-496d725a2c4a" (UID: "4a4a131c-5e02-4a28-934e-496d725a2c4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 14:02:18.320064 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.320043 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a4a131c-5e02-4a28-934e-496d725a2c4a" (UID: "4a4a131c-5e02-4a28-934e-496d725a2c4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:18.320064 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.320053 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-kube-api-access-6q9jv" (OuterVolumeSpecName: "kube-api-access-6q9jv") pod "4a4a131c-5e02-4a28-934e-496d725a2c4a" (UID: "4a4a131c-5e02-4a28-934e-496d725a2c4a"). InnerVolumeSpecName "kube-api-access-6q9jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:02:18.353378 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.353343 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerID="7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec" exitCode=0 Jul 07 14:02:18.353569 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.353406 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" Jul 07 14:02:18.353569 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.353420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" event={"ID":"4a4a131c-5e02-4a28-934e-496d725a2c4a","Type":"ContainerDied","Data":"7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec"} Jul 07 14:02:18.353569 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.353501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562" event={"ID":"4a4a131c-5e02-4a28-934e-496d725a2c4a","Type":"ContainerDied","Data":"a54d40e635a0b5dedef1c080aa0d2665aa457b29b7b24184f328d95fd9ec3498"} Jul 07 14:02:18.353569 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.353528 2576 scope.go:117] "RemoveContainer" containerID="7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec" Jul 07 14:02:18.363219 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.363203 2576 scope.go:117] "RemoveContainer" containerID="eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665" Jul 07 14:02:18.370984 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.370959 2576 scope.go:117] "RemoveContainer" containerID="7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec" Jul 07 14:02:18.371282 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:02:18.371261 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec\": container with ID starting with 7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec not found: ID does not exist" containerID="7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec" Jul 07 14:02:18.371347 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.371291 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec"} err="failed to get container status \"7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec\": rpc error: code = NotFound desc = could not find container \"7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec\": container with ID starting with 7db68c571c60aafa7794a3c40934e28b70ec7d6e67aae57261118c08933cf9ec not found: ID does not exist" Jul 07 14:02:18.371347 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.371308 2576 scope.go:117] "RemoveContainer" containerID="eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665" Jul 07 14:02:18.371557 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:02:18.371540 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665\": container with ID starting with eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665 not found: ID does not exist" containerID="eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665" Jul 07 14:02:18.371602 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.371562 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665"} err="failed to get container status \"eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665\": rpc error: code = NotFound desc = could not find container \"eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665\": container with ID starting with eb95f03e223dda4780099f0c7637bcefd802f812e88244dbfd430ac71b78a665 not found: ID does not exist" Jul 07 14:02:18.379269 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.379248 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562"] Jul 07 14:02:18.392411 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.392386 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-da5db0-openshift-default-64f8dd5bc-pp562"] Jul 07 14:02:18.418635 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.418605 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:18.418635 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.418633 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4a131c-5e02-4a28-934e-496d725a2c4a-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:18.418863 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.418649 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:18.418863 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.418659 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:18.418863 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.418667 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:18.418863 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.418675 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:18.418863 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.418683 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:18.418863 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.418691 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q9jv\" (UniqueName: \"kubernetes.io/projected/4a4a131c-5e02-4a28-934e-496d725a2c4a-kube-api-access-6q9jv\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:18.418863 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.418699 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4a131c-5e02-4a28-934e-496d725a2c4a-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:18.589973 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:18.589897 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" path="/var/lib/kubelet/pods/4a4a131c-5e02-4a28-934e-496d725a2c4a/volumes" Jul 07 14:02:22.822408 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.822372 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2"] Jul 07 14:02:22.822769 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.822752 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" Jul 07 14:02:22.822769 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.822763 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" Jul 07 14:02:22.822852 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.822773 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" Jul 07 14:02:22.822852 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.822780 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" Jul 07 14:02:22.822917 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.822872 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf1c722a-a9ac-4ad1-83e9-aa099fa26488" containerName="istio-proxy" Jul 07 14:02:22.822917 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.822881 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" Jul 07 14:02:22.825535 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.825515 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.836733 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.836714 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-db9337-openshift-default-dockercfg-9jh2s\"" Jul 07 14:02:22.852606 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.852574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2"] Jul 07 14:02:22.859717 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.859691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-data\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.859839 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.859740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbcqm\" (UniqueName: \"kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-kube-api-access-xbcqm\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.859839 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.859764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-token\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.859967 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.859831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-socket\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.859967 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.859902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-podinfo\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.859967 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.859936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-certs\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.859967 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.859961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-credential-socket\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.860202 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.859978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-envoy\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.860202 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.860012 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istiod-ca-cert\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.960974 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.960937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbcqm\" (UniqueName: \"kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-kube-api-access-xbcqm\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.960974 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.960976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-token\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961254 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.960996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-socket\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961254 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.961028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-podinfo\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961254 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.961050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-certs\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961254 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.961070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-credential-socket\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961254 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.961088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-envoy\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961254 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.961107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istiod-ca-cert\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961254 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.961195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-data\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961605 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.961530 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-socket\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961704 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.961680 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-certs\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961770 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.961749 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-credential-socket\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.961827 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.961772 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-data\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.962106 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.962083 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istiod-ca-cert\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.963464 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.963442 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-envoy\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.963683 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.963667 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-podinfo\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:22.988627 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:22.988595 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbcqm\" (UniqueName: \"kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-kube-api-access-xbcqm\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:23.008453 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:23.008426 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-token\") pod \"e2e-api-b-db9337-openshift-default-674c596f85-rfzc2\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:23.135099 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:23.135027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:23.285745 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:23.285718 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2"] Jul 07 14:02:23.287411 ip-10-0-132-236 kubenswrapper[2576]: W0707 14:02:23.287382 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77aa0b3b_0f2c_48f2_972c_2a95cfde92fa.slice/crio-777b1d45d310fac235ab39b07d9f0dd041fa6180aae03c71adca967b0a034bb6 WatchSource:0}: Error finding container 777b1d45d310fac235ab39b07d9f0dd041fa6180aae03c71adca967b0a034bb6: Status 404 returned error can't find the container with id 777b1d45d310fac235ab39b07d9f0dd041fa6180aae03c71adca967b0a034bb6 Jul 07 14:02:23.289552 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:23.289519 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:23.289637 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:23.289585 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:23.289637 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:23.289613 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:23.380518 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:23.380485 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" event={"ID":"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa","Type":"ContainerStarted","Data":"e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d"} Jul 07 14:02:23.380664 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:23.380544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" event={"ID":"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa","Type":"ContainerStarted","Data":"777b1d45d310fac235ab39b07d9f0dd041fa6180aae03c71adca967b0a034bb6"} Jul 07 14:02:23.434013 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:23.433907 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" podStartSLOduration=1.433892412 podStartE2EDuration="1.433892412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:02:23.433192414 +0000 UTC m=+2077.434433946" watchObservedRunningTime="2026-07-07 14:02:23.433892412 +0000 UTC m=+2077.435133938" Jul 07 14:02:24.135389 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:24.135357 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:24.140579 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:24.140555 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:24.385740 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:24.385658 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:24.386673 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:24.386649 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:31.951476 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:31.950142 2576 patch_prober.go:28] interesting pod/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": read tcp 10.134.0.2:37004->10.134.0.63:15021: read: connection reset by peer" start-of-body= Jul 07 14:02:31.951476 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:31.950234 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": read tcp 10.134.0.2:37004->10.134.0.63:15021: read: connection reset by peer" Jul 07 14:02:32.428100 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:32.428067 2576 generic.go:356] "Generic (PLEG): container finished" podID="d58d742c-063d-4f71-a722-da67acc84a83" containerID="186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8" exitCode=0 Jul 07 14:02:32.428299 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:32.428141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" event={"ID":"d58d742c-063d-4f71-a722-da67acc84a83","Type":"ContainerDied","Data":"186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8"} Jul 07 14:02:32.428577 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:32.428561 2576 scope.go:117] "RemoveContainer" containerID="186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8" Jul 07 14:02:32.429589 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:32.429557 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:32.429650 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:32.429629 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:32.429690 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:32.429660 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:33.436356 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:33.436325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" event={"ID":"d58d742c-063d-4f71-a722-da67acc84a83","Type":"ContainerStarted","Data":"8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97"} Jul 07 14:02:34.250796 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:34.250762 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:34.251119 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:34.251091 2576 patch_prober.go:28] interesting pod/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 07 14:02:34.251274 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:34.251239 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 07 14:02:35.252603 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:35.252424 2576 patch_prober.go:28] interesting pod/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 07 14:02:35.253587 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:35.253554 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 07 14:02:36.251635 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:36.251593 2576 patch_prober.go:28] interesting pod/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 07 14:02:36.252025 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:36.251995 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 07 14:02:37.251450 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:37.251407 2576 patch_prober.go:28] interesting pod/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 07 14:02:37.251965 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:37.251478 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 07 14:02:38.254785 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:38.254744 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:38.255305 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:38.255282 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:38.256130 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:38.256102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:40.427376 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:40.427327 2576 patch_prober.go:28] interesting pod/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 14:02:40.427793 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:40.427410 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 14:02:41.496834 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:41.496801 2576 generic.go:356] "Generic (PLEG): container finished" podID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerID="e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d" exitCode=0 Jul 07 14:02:41.497278 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:41.496853 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" event={"ID":"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa","Type":"ContainerDied","Data":"e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d"} Jul 07 14:02:41.497278 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:41.497234 2576 scope.go:117] "RemoveContainer" containerID="e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d" Jul 07 14:02:41.498345 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:41.498308 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:41.498461 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:41.498377 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:41.498461 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:41.498407 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:42.503897 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:42.503863 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" event={"ID":"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa","Type":"ContainerStarted","Data":"ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71"} Jul 07 14:02:43.136138 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:43.136102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:43.136380 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:43.136356 2576 patch_prober.go:28] interesting pod/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 07 14:02:43.136450 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:43.136433 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 07 14:02:44.135819 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:44.135783 2576 patch_prober.go:28] interesting pod/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 07 14:02:44.136225 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:44.135839 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 07 14:02:45.136244 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:45.136204 2576 patch_prober.go:28] interesting pod/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 07 14:02:45.136709 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:45.136266 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 07 14:02:46.135773 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:46.135736 2576 patch_prober.go:28] interesting pod/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 07 14:02:46.135952 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:46.135787 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 07 14:02:47.139129 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:47.139091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:47.139515 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:47.139390 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:47.139999 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:47.139985 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:02:48.969590 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:48.969557 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x"] Jul 07 14:02:48.970020 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:48.969750 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" containerID="cri-o://8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97" gracePeriod=30 Jul 07 14:02:53.257238 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:53.257197 2576 patch_prober.go:28] interesting pod/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 14:02:53.257631 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:53.257272 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:02:54.222691 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.222665 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:54.386061 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.385972 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-socket\") pod \"d58d742c-063d-4f71-a722-da67acc84a83\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " Jul 07 14:02:54.386061 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386012 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-credential-socket\") pod \"d58d742c-063d-4f71-a722-da67acc84a83\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " Jul 07 14:02:54.386061 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386041 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58d742c-063d-4f71-a722-da67acc84a83-istio-podinfo\") pod \"d58d742c-063d-4f71-a722-da67acc84a83\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " Jul 07 14:02:54.386655 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386091 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-data\") pod \"d58d742c-063d-4f71-a722-da67acc84a83\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " Jul 07 14:02:54.386655 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386131 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlj7d\" (UniqueName: \"kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-kube-api-access-jlj7d\") pod \"d58d742c-063d-4f71-a722-da67acc84a83\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " Jul 07 14:02:54.386655 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386193 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-istio-token\") pod \"d58d742c-063d-4f71-a722-da67acc84a83\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " Jul 07 14:02:54.386655 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386227 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-certs\") pod \"d58d742c-063d-4f71-a722-da67acc84a83\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " Jul 07 14:02:54.386655 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386289 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d58d742c-063d-4f71-a722-da67acc84a83" (UID: "d58d742c-063d-4f71-a722-da67acc84a83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:54.386655 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386312 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d58d742c-063d-4f71-a722-da67acc84a83" (UID: "d58d742c-063d-4f71-a722-da67acc84a83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:54.386655 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386403 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58d742c-063d-4f71-a722-da67acc84a83-istiod-ca-cert\") pod \"d58d742c-063d-4f71-a722-da67acc84a83\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " Jul 07 14:02:54.386655 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386439 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-envoy\") pod \"d58d742c-063d-4f71-a722-da67acc84a83\" (UID: \"d58d742c-063d-4f71-a722-da67acc84a83\") " Jul 07 14:02:54.386655 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386525 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d58d742c-063d-4f71-a722-da67acc84a83" (UID: "d58d742c-063d-4f71-a722-da67acc84a83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:54.387036 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386745 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58d742c-063d-4f71-a722-da67acc84a83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d58d742c-063d-4f71-a722-da67acc84a83" (UID: "d58d742c-063d-4f71-a722-da67acc84a83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:02:54.387036 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386756 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:54.387036 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386776 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:54.387036 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386790 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:54.387036 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.386963 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-data" (OuterVolumeSpecName: "istio-data") pod "d58d742c-063d-4f71-a722-da67acc84a83" (UID: "d58d742c-063d-4f71-a722-da67acc84a83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:54.388401 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.388381 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-istio-token" (OuterVolumeSpecName: "istio-token") pod "d58d742c-063d-4f71-a722-da67acc84a83" (UID: "d58d742c-063d-4f71-a722-da67acc84a83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:02:54.388475 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.388386 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d58d742c-063d-4f71-a722-da67acc84a83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d58d742c-063d-4f71-a722-da67acc84a83" (UID: "d58d742c-063d-4f71-a722-da67acc84a83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 14:02:54.388668 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.388650 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-kube-api-access-jlj7d" (OuterVolumeSpecName: "kube-api-access-jlj7d") pod "d58d742c-063d-4f71-a722-da67acc84a83" (UID: "d58d742c-063d-4f71-a722-da67acc84a83"). InnerVolumeSpecName "kube-api-access-jlj7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:02:54.388725 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.388683 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d58d742c-063d-4f71-a722-da67acc84a83" (UID: "d58d742c-063d-4f71-a722-da67acc84a83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:02:54.488015 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.487972 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58d742c-063d-4f71-a722-da67acc84a83-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:54.488015 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.488008 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:54.488015 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.488017 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlj7d\" (UniqueName: \"kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-kube-api-access-jlj7d\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:54.488015 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.488027 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58d742c-063d-4f71-a722-da67acc84a83-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:54.488330 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.488036 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58d742c-063d-4f71-a722-da67acc84a83-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:54.488330 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.488044 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58d742c-063d-4f71-a722-da67acc84a83-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:02:54.563551 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.563515 2576 generic.go:356] "Generic (PLEG): container finished" podID="d58d742c-063d-4f71-a722-da67acc84a83" containerID="8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97" exitCode=0 Jul 07 14:02:54.563719 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.563557 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" Jul 07 14:02:54.563719 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.563596 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" event={"ID":"d58d742c-063d-4f71-a722-da67acc84a83","Type":"ContainerDied","Data":"8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97"} Jul 07 14:02:54.563719 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.563658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x" event={"ID":"d58d742c-063d-4f71-a722-da67acc84a83","Type":"ContainerDied","Data":"33a12eef6950a7cef8e8c1ae2a051ceb3f90fda85e8e6c01b98d7a239079bf1c"} Jul 07 14:02:54.563719 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.563685 2576 scope.go:117] "RemoveContainer" containerID="8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97" Jul 07 14:02:54.573540 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.573518 2576 scope.go:117] "RemoveContainer" containerID="186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8" Jul 07 14:02:54.581442 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.581421 2576 scope.go:117] "RemoveContainer" containerID="8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97" Jul 07 14:02:54.581720 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:02:54.581697 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97\": container with ID starting with 8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97 not found: ID does not exist" containerID="8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97" Jul 07 14:02:54.581795 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.581729 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97"} err="failed to get container status \"8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97\": rpc error: code = NotFound desc = could not find container \"8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97\": container with ID starting with 8aa985fd5cdce2e1d536bd6e4458e9d41a57887b1a0f480c400ce900d5bb5f97 not found: ID does not exist" Jul 07 14:02:54.581795 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.581747 2576 scope.go:117] "RemoveContainer" containerID="186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8" Jul 07 14:02:54.581979 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:02:54.581965 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8\": container with ID starting with 186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8 not found: ID does not exist" containerID="186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8" Jul 07 14:02:54.582022 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.581983 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8"} err="failed to get container status \"186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8\": rpc error: code = NotFound desc = could not find container \"186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8\": container with ID starting with 186fdf859cf026db79708f30ec802ab117b584be4375b280af64059f8d2918f8 not found: ID does not exist" Jul 07 14:02:54.597018 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.596990 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x"] Jul 07 14:02:54.603804 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:54.603781 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d1d8a6-openshift-default-5b56b98d4c-vvn4x"] Jul 07 14:02:56.590057 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:56.590027 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d58d742c-063d-4f71-a722-da67acc84a83" path="/var/lib/kubelet/pods/d58d742c-063d-4f71-a722-da67acc84a83/volumes" Jul 07 14:02:57.727201 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:57.727156 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2"] Jul 07 14:02:57.727608 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:57.727348 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" containerID="cri-o://ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71" gracePeriod=30 Jul 07 14:02:58.698999 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.698969 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh"] Jul 07 14:02:58.699368 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.699355 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" Jul 07 14:02:58.699420 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.699370 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" Jul 07 14:02:58.699465 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.699435 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a4a131c-5e02-4a28-934e-496d725a2c4a" containerName="istio-proxy" Jul 07 14:02:58.699465 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.699445 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" Jul 07 14:02:58.699465 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.699451 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" Jul 07 14:02:58.702656 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.702642 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.705797 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.705777 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-b6630e-openshift-default-dockercfg-wc6rl\"" Jul 07 14:02:58.721702 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.721677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjgrt\" (UniqueName: \"kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-kube-api-access-pjgrt\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.721820 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.721714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-credential-socket\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.721820 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.721747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-socket\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.721820 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.721766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-podinfo\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.721820 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.721787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-token\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.721820 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.721808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-envoy\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.721992 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.721860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f93c2b-9cc2-45f2-978b-9795e63423ac-istiod-ca-cert\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.721992 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.721903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-certs\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.721992 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.721923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-data\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.725872 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.725845 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh"] Jul 07 14:02:58.822785 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.822752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-socket\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823236 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.822789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-podinfo\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823236 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.822887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-token\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823236 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.822907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-envoy\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823236 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.822931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f93c2b-9cc2-45f2-978b-9795e63423ac-istiod-ca-cert\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823236 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.822964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-certs\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823236 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.822992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-data\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823236 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.823064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjgrt\" (UniqueName: \"kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-kube-api-access-pjgrt\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823236 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.823103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-credential-socket\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823643 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.823240 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-socket\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823643 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.823406 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-certs\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823643 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.823450 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-credential-socket\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823643 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.823497 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-data\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.823842 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.823757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f93c2b-9cc2-45f2-978b-9795e63423ac-istiod-ca-cert\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.825243 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.825220 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-envoy\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.825473 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.825458 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-podinfo\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.833426 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.833374 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-token\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:58.834131 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:58.834109 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjgrt\" (UniqueName: \"kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-kube-api-access-pjgrt\") pod \"e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:59.012679 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:59.012643 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:02:59.166276 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:59.166251 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh"] Jul 07 14:02:59.168665 ip-10-0-132-236 kubenswrapper[2576]: W0707 14:02:59.168642 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f93c2b_9cc2_45f2_978b_9795e63423ac.slice/crio-1d4b7e4cb2b0c2e2daeba47baea209bc12c86a3f17149dbcfc5fb6af98b7e5a9 WatchSource:0}: Error finding container 1d4b7e4cb2b0c2e2daeba47baea209bc12c86a3f17149dbcfc5fb6af98b7e5a9: Status 404 returned error can't find the container with id 1d4b7e4cb2b0c2e2daeba47baea209bc12c86a3f17149dbcfc5fb6af98b7e5a9 Jul 07 14:02:59.170819 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:59.170786 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:59.170903 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:59.170846 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:59.170903 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:59.170895 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:02:59.590677 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:59.590642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" event={"ID":"06f93c2b-9cc2-45f2-978b-9795e63423ac","Type":"ContainerStarted","Data":"66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6"} Jul 07 14:02:59.590831 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:59.590694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" event={"ID":"06f93c2b-9cc2-45f2-978b-9795e63423ac","Type":"ContainerStarted","Data":"1d4b7e4cb2b0c2e2daeba47baea209bc12c86a3f17149dbcfc5fb6af98b7e5a9"} Jul 07 14:02:59.632805 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:02:59.632759 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" podStartSLOduration=1.632744194 podStartE2EDuration="1.632744194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:02:59.631953406 +0000 UTC m=+2113.633194932" watchObservedRunningTime="2026-07-07 14:02:59.632744194 +0000 UTC m=+2113.633985720" Jul 07 14:03:00.013237 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:00.013203 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:03:00.018158 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:00.018136 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:03:00.595840 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:00.595811 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:03:00.596869 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:00.596849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:03:02.142082 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:02.142047 2576 patch_prober.go:28] interesting pod/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 14:03:02.142459 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:02.142106 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:03:02.990765 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:02.990693 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:03:03.059098 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059066 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-socket\") pod \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " Jul 07 14:03:03.059320 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059124 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-credential-socket\") pod \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " Jul 07 14:03:03.059320 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059175 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbcqm\" (UniqueName: \"kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-kube-api-access-xbcqm\") pod \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " Jul 07 14:03:03.059320 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059205 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istiod-ca-cert\") pod \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " Jul 07 14:03:03.059320 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059256 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-envoy\") pod \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " Jul 07 14:03:03.059320 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059282 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-token\") pod \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " Jul 07 14:03:03.059320 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059310 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-certs\") pod \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " Jul 07 14:03:03.059574 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059361 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-podinfo\") pod \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " Jul 07 14:03:03.059574 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059388 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" (UID: "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:03:03.059574 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059402 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-data\") pod \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\" (UID: \"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa\") " Jul 07 14:03:03.059574 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059426 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" (UID: "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:03:03.059784 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059739 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" (UID: "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:03:03.059784 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059741 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:03:03.059784 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059775 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:03:03.059934 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.059750 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" (UID: "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:03:03.060309 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.060284 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" (UID: "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:03:03.061773 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.061749 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" (UID: "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 14:03:03.062025 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.062005 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-kube-api-access-xbcqm" (OuterVolumeSpecName: "kube-api-access-xbcqm") pod "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" (UID: "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa"). InnerVolumeSpecName "kube-api-access-xbcqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:03:03.062194 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.062153 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" (UID: "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:03:03.062308 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.062226 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" (UID: "77aa0b3b-0f2c-48f2-972c-2a95cfde92fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:03:03.160841 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.160806 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:03:03.160841 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.160836 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:03:03.160841 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.160845 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbcqm\" (UniqueName: \"kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-kube-api-access-xbcqm\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:03:03.161310 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.160855 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:03:03.161310 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.160864 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:03:03.161310 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.160872 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:03:03.161310 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.160880 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:03:03.611749 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.611715 2576 generic.go:356] "Generic (PLEG): container finished" podID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerID="ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71" exitCode=0 Jul 07 14:03:03.611963 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.611771 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" Jul 07 14:03:03.611963 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.611820 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" event={"ID":"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa","Type":"ContainerDied","Data":"ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71"} Jul 07 14:03:03.611963 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.611875 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2" event={"ID":"77aa0b3b-0f2c-48f2-972c-2a95cfde92fa","Type":"ContainerDied","Data":"777b1d45d310fac235ab39b07d9f0dd041fa6180aae03c71adca967b0a034bb6"} Jul 07 14:03:03.611963 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.611905 2576 scope.go:117] "RemoveContainer" containerID="ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71" Jul 07 14:03:03.623240 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.623219 2576 scope.go:117] "RemoveContainer" containerID="e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d" Jul 07 14:03:03.631396 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.631378 2576 scope.go:117] "RemoveContainer" containerID="ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71" Jul 07 14:03:03.631653 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:03:03.631634 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71\": container with ID starting with ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71 not found: ID does not exist" containerID="ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71" Jul 07 14:03:03.631724 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.631662 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71"} err="failed to get container status \"ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71\": rpc error: code = NotFound desc = could not find container \"ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71\": container with ID starting with ba7fe814f986b45cb975d854cc0b101e8bf2d71cb2971ef77f8ce05fe1c51e71 not found: ID does not exist" Jul 07 14:03:03.631724 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.631679 2576 scope.go:117] "RemoveContainer" containerID="e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d" Jul 07 14:03:03.631942 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:03:03.631919 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d\": container with ID starting with e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d not found: ID does not exist" containerID="e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d" Jul 07 14:03:03.631983 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.631951 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d"} err="failed to get container status \"e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d\": rpc error: code = NotFound desc = could not find container \"e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d\": container with ID starting with e7ac78a6af3a7676da63e2b3a5eae3e586e46643cef20f0b3575f7fdca7ede4d not found: ID does not exist" Jul 07 14:03:03.655278 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.655248 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2"] Jul 07 14:03:03.662335 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:03.662310 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-db9337-openshift-default-674c596f85-rfzc2"] Jul 07 14:03:04.590872 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:04.590827 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" path="/var/lib/kubelet/pods/77aa0b3b-0f2c-48f2-972c-2a95cfde92fa/volumes" Jul 07 14:03:07.323898 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.323854 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9"] Jul 07 14:03:07.324416 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.324257 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" Jul 07 14:03:07.324416 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.324271 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" Jul 07 14:03:07.324416 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.324287 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" Jul 07 14:03:07.324416 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.324293 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d58d742c-063d-4f71-a722-da67acc84a83" containerName="istio-proxy" Jul 07 14:03:07.324416 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.324373 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" Jul 07 14:03:07.324416 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.324384 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" Jul 07 14:03:07.327355 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.327338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.340317 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.340289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-f0b1df-openshift-default-dockercfg-jsvhl\"" Jul 07 14:03:07.368438 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.368410 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9"] Jul 07 14:03:07.402268 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.402235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-socket\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.402412 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.402282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-data\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.402412 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.402326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda375d4-1c20-4ada-81fc-6528a6035a69-istiod-ca-cert\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.402412 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.402381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-certs\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.402412 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.402400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xddd\" (UniqueName: \"kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-kube-api-access-7xddd\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.402575 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.402418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-podinfo\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.402575 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.402510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-token\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.402660 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.402575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-credential-socket\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.402660 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.402607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-envoy\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.503968 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.503935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-credential-socket\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.503968 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.503966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-envoy\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504222 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-socket\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504222 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504037 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-data\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504222 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda375d4-1c20-4ada-81fc-6528a6035a69-istiod-ca-cert\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504222 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-certs\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504222 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xddd\" (UniqueName: \"kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-kube-api-access-7xddd\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504222 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-podinfo\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504222 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-token\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504601 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504417 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-credential-socket\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504601 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-socket\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504601 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504523 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-certs\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504775 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504663 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-data\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.504864 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.504844 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda375d4-1c20-4ada-81fc-6528a6035a69-istiod-ca-cert\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.506432 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.506408 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-envoy\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.506589 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.506573 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-podinfo\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.521503 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.521481 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xddd\" (UniqueName: \"kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-kube-api-access-7xddd\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.523014 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.522998 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-token\") pod \"e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.641297 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.641209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:07.806000 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.805976 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9"] Jul 07 14:03:07.808123 ip-10-0-132-236 kubenswrapper[2576]: W0707 14:03:07.808099 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda375d4_1c20_4ada_81fc_6528a6035a69.slice/crio-8f0452aff58aba92e4a85d0e6b559911aa7333a0c1b18b4c4618ec260c962e6c WatchSource:0}: Error finding container 8f0452aff58aba92e4a85d0e6b559911aa7333a0c1b18b4c4618ec260c962e6c: Status 404 returned error can't find the container with id 8f0452aff58aba92e4a85d0e6b559911aa7333a0c1b18b4c4618ec260c962e6c Jul 07 14:03:07.810290 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.810258 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:03:07.810380 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.810323 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:03:07.810380 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:07.810353 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:03:08.639434 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:08.639395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" event={"ID":"fda375d4-1c20-4ada-81fc-6528a6035a69","Type":"ContainerStarted","Data":"4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932"} Jul 07 14:03:08.639821 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:08.639456 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" event={"ID":"fda375d4-1c20-4ada-81fc-6528a6035a69","Type":"ContainerStarted","Data":"8f0452aff58aba92e4a85d0e6b559911aa7333a0c1b18b4c4618ec260c962e6c"} Jul 07 14:03:08.641816 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:08.641797 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:08.646517 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:08.646496 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:08.697204 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:08.697084 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" podStartSLOduration=1.697069771 podStartE2EDuration="1.697069771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:03:08.696730674 +0000 UTC m=+2122.697972213" watchObservedRunningTime="2026-07-07 14:03:08.697069771 +0000 UTC m=+2122.698311298" Jul 07 14:03:09.644889 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:09.644859 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:09.645844 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:09.645825 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:16.638325 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:16.638286 2576 patch_prober.go:28] interesting pod/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 14:03:16.639710 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:16.638355 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 14:03:17.694365 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:17.694331 2576 generic.go:356] "Generic (PLEG): container finished" podID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerID="66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6" exitCode=0 Jul 07 14:03:17.694752 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:17.694410 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" event={"ID":"06f93c2b-9cc2-45f2-978b-9795e63423ac","Type":"ContainerDied","Data":"66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6"} Jul 07 14:03:17.694811 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:17.694802 2576 scope.go:117] "RemoveContainer" containerID="66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6" Jul 07 14:03:17.695839 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:17.695812 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:03:17.695911 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:17.695882 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:03:17.695993 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:17.695915 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:03:18.701093 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:18.701060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" event={"ID":"06f93c2b-9cc2-45f2-978b-9795e63423ac","Type":"ContainerStarted","Data":"11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c"} Jul 07 14:03:19.013145 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:19.013103 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:03:19.013481 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:19.013458 2576 patch_prober.go:28] interesting pod/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 07 14:03:19.013538 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:19.013517 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 07 14:03:20.013693 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:20.013663 2576 patch_prober.go:28] interesting pod/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 07 14:03:20.014204 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:20.014149 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 07 14:03:21.013575 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:21.013537 2576 patch_prober.go:28] interesting pod/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 07 14:03:21.013753 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:21.013590 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 07 14:03:22.013216 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:22.013185 2576 patch_prober.go:28] interesting pod/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 07 14:03:22.013461 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:22.013433 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 07 14:03:23.016715 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:23.016679 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:03:23.017110 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:23.017070 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:03:23.017671 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:23.017652 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:03:25.202664 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:25.202422 2576 patch_prober.go:28] interesting pod/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": read tcp 10.134.0.2:49082->10.134.0.66:15021: read: connection reset by peer" start-of-body= Jul 07 14:03:25.202664 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:25.202489 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" podUID="fda375d4-1c20-4ada-81fc-6528a6035a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": read tcp 10.134.0.2:49082->10.134.0.66:15021: read: connection reset by peer" Jul 07 14:03:25.745622 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:25.745588 2576 generic.go:356] "Generic (PLEG): container finished" podID="fda375d4-1c20-4ada-81fc-6528a6035a69" containerID="4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932" exitCode=0 Jul 07 14:03:25.745792 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:25.745660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" event={"ID":"fda375d4-1c20-4ada-81fc-6528a6035a69","Type":"ContainerDied","Data":"4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932"} Jul 07 14:03:25.746064 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:25.746051 2576 scope.go:117] "RemoveContainer" containerID="4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932" Jul 07 14:03:25.747068 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:25.747042 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:03:25.747131 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:25.747110 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:03:25.747189 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:25.747152 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 14:03:26.752661 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:26.752621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" event={"ID":"fda375d4-1c20-4ada-81fc-6528a6035a69","Type":"ContainerStarted","Data":"658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378"} Jul 07 14:03:27.642120 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:27.642078 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:27.642440 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:27.642418 2576 patch_prober.go:28] interesting pod/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 07 14:03:27.642499 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:27.642467 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" podUID="fda375d4-1c20-4ada-81fc-6528a6035a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 07 14:03:28.641921 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:28.641886 2576 patch_prober.go:28] interesting pod/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 07 14:03:28.642336 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:28.641947 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" podUID="fda375d4-1c20-4ada-81fc-6528a6035a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 07 14:03:29.642012 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:29.641979 2576 patch_prober.go:28] interesting pod/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 07 14:03:29.642420 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:29.642030 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" podUID="fda375d4-1c20-4ada-81fc-6528a6035a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 07 14:03:30.644836 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:30.644803 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:30.645261 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:30.645068 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:03:30.645871 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:03:30.645855 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:08:57.386861 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:08:57.386829 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh"] Jul 07 14:08:57.388344 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:08:57.387074 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" containerID="cri-o://11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c" gracePeriod=30 Jul 07 14:09:02.648377 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.648351 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:09:02.807592 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.807549 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-credential-socket\") pod \"06f93c2b-9cc2-45f2-978b-9795e63423ac\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " Jul 07 14:09:02.807792 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.807616 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjgrt\" (UniqueName: \"kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-kube-api-access-pjgrt\") pod \"06f93c2b-9cc2-45f2-978b-9795e63423ac\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " Jul 07 14:09:02.807792 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.807644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-socket\") pod \"06f93c2b-9cc2-45f2-978b-9795e63423ac\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " Jul 07 14:09:02.807792 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.807757 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f93c2b-9cc2-45f2-978b-9795e63423ac-istiod-ca-cert\") pod \"06f93c2b-9cc2-45f2-978b-9795e63423ac\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " Jul 07 14:09:02.807976 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.807863 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-data\") pod \"06f93c2b-9cc2-45f2-978b-9795e63423ac\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " Jul 07 14:09:02.807976 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.807896 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "06f93c2b-9cc2-45f2-978b-9795e63423ac" (UID: "06f93c2b-9cc2-45f2-978b-9795e63423ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:09:02.807976 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.807907 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "06f93c2b-9cc2-45f2-978b-9795e63423ac" (UID: "06f93c2b-9cc2-45f2-978b-9795e63423ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:09:02.807976 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.807923 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-envoy\") pod \"06f93c2b-9cc2-45f2-978b-9795e63423ac\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " Jul 07 14:09:02.807976 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.807957 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-podinfo\") pod \"06f93c2b-9cc2-45f2-978b-9795e63423ac\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " Jul 07 14:09:02.808268 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.807992 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-token\") pod \"06f93c2b-9cc2-45f2-978b-9795e63423ac\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " Jul 07 14:09:02.808268 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.808016 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-certs\") pod \"06f93c2b-9cc2-45f2-978b-9795e63423ac\" (UID: \"06f93c2b-9cc2-45f2-978b-9795e63423ac\") " Jul 07 14:09:02.808268 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.808188 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f93c2b-9cc2-45f2-978b-9795e63423ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "06f93c2b-9cc2-45f2-978b-9795e63423ac" (UID: "06f93c2b-9cc2-45f2-978b-9795e63423ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:09:02.808483 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.808448 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "06f93c2b-9cc2-45f2-978b-9795e63423ac" (UID: "06f93c2b-9cc2-45f2-978b-9795e63423ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:09:02.808555 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.808531 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:02.808555 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.808548 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f93c2b-9cc2-45f2-978b-9795e63423ac-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:02.808653 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.808564 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:02.808653 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.808577 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:02.808788 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.808765 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "06f93c2b-9cc2-45f2-978b-9795e63423ac" (UID: "06f93c2b-9cc2-45f2-978b-9795e63423ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:09:02.810041 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.810018 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "06f93c2b-9cc2-45f2-978b-9795e63423ac" (UID: "06f93c2b-9cc2-45f2-978b-9795e63423ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:09:02.810146 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.810039 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "06f93c2b-9cc2-45f2-978b-9795e63423ac" (UID: "06f93c2b-9cc2-45f2-978b-9795e63423ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 14:09:02.810330 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.810315 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-kube-api-access-pjgrt" (OuterVolumeSpecName: "kube-api-access-pjgrt") pod "06f93c2b-9cc2-45f2-978b-9795e63423ac" (UID: "06f93c2b-9cc2-45f2-978b-9795e63423ac"). InnerVolumeSpecName "kube-api-access-pjgrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:09:02.810395 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.810348 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "06f93c2b-9cc2-45f2-978b-9795e63423ac" (UID: "06f93c2b-9cc2-45f2-978b-9795e63423ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:09:02.909444 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.909397 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:02.909444 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.909438 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:02.909444 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.909448 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:02.909444 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.909457 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:02.909733 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:02.909467 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjgrt\" (UniqueName: \"kubernetes.io/projected/06f93c2b-9cc2-45f2-978b-9795e63423ac-kube-api-access-pjgrt\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:03.418638 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.418602 2576 generic.go:356] "Generic (PLEG): container finished" podID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerID="11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c" exitCode=0 Jul 07 14:09:03.418845 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.418663 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" event={"ID":"06f93c2b-9cc2-45f2-978b-9795e63423ac","Type":"ContainerDied","Data":"11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c"} Jul 07 14:09:03.418845 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.418668 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" Jul 07 14:09:03.418845 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.418704 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh" event={"ID":"06f93c2b-9cc2-45f2-978b-9795e63423ac","Type":"ContainerDied","Data":"1d4b7e4cb2b0c2e2daeba47baea209bc12c86a3f17149dbcfc5fb6af98b7e5a9"} Jul 07 14:09:03.418845 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.418726 2576 scope.go:117] "RemoveContainer" containerID="11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c" Jul 07 14:09:03.428816 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.428795 2576 scope.go:117] "RemoveContainer" containerID="66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6" Jul 07 14:09:03.436912 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.436891 2576 scope.go:117] "RemoveContainer" containerID="11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c" Jul 07 14:09:03.437200 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:09:03.437161 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c\": container with ID starting with 11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c not found: ID does not exist" containerID="11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c" Jul 07 14:09:03.437260 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.437212 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c"} err="failed to get container status \"11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c\": rpc error: code = NotFound desc = could not find container \"11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c\": container with ID starting with 11e26521a5741c691905a77b54ae649ac552c4598789a46959b88ef6325dc44c not found: ID does not exist" Jul 07 14:09:03.437260 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.437235 2576 scope.go:117] "RemoveContainer" containerID="66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6" Jul 07 14:09:03.437492 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:09:03.437469 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6\": container with ID starting with 66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6 not found: ID does not exist" containerID="66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6" Jul 07 14:09:03.437553 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.437502 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6"} err="failed to get container status \"66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6\": rpc error: code = NotFound desc = could not find container \"66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6\": container with ID starting with 66efc6e817b11d0bda6829226f8dc983d5bb3de179bbfd0d98687fc77e707da6 not found: ID does not exist" Jul 07 14:09:03.443659 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.443636 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh"] Jul 07 14:09:03.448540 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:03.448519 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-b6630e-openshift-default-7c6b9fdff5-q6hxh"] Jul 07 14:09:04.590644 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:04.590610 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" path="/var/lib/kubelet/pods/06f93c2b-9cc2-45f2-978b-9795e63423ac/volumes" Jul 07 14:09:04.772049 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:04.772010 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9"] Jul 07 14:09:04.772277 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:04.772254 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" podUID="fda375d4-1c20-4ada-81fc-6528a6035a69" containerName="istio-proxy" containerID="cri-o://658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378" gracePeriod=30 Jul 07 14:09:10.029333 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.029312 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:09:10.176338 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176261 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xddd\" (UniqueName: \"kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-kube-api-access-7xddd\") pod \"fda375d4-1c20-4ada-81fc-6528a6035a69\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " Jul 07 14:09:10.176338 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176314 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-credential-socket\") pod \"fda375d4-1c20-4ada-81fc-6528a6035a69\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " Jul 07 14:09:10.176552 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176377 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-certs\") pod \"fda375d4-1c20-4ada-81fc-6528a6035a69\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " Jul 07 14:09:10.176552 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176413 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-token\") pod \"fda375d4-1c20-4ada-81fc-6528a6035a69\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " Jul 07 14:09:10.176552 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176446 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-data\") pod \"fda375d4-1c20-4ada-81fc-6528a6035a69\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " Jul 07 14:09:10.176552 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176514 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-podinfo\") pod \"fda375d4-1c20-4ada-81fc-6528a6035a69\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " Jul 07 14:09:10.176552 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176544 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda375d4-1c20-4ada-81fc-6528a6035a69-istiod-ca-cert\") pod \"fda375d4-1c20-4ada-81fc-6528a6035a69\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " Jul 07 14:09:10.176817 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176575 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-envoy\") pod \"fda375d4-1c20-4ada-81fc-6528a6035a69\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " Jul 07 14:09:10.176817 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176609 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-socket\") pod \"fda375d4-1c20-4ada-81fc-6528a6035a69\" (UID: \"fda375d4-1c20-4ada-81fc-6528a6035a69\") " Jul 07 14:09:10.176817 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176634 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fda375d4-1c20-4ada-81fc-6528a6035a69" (UID: "fda375d4-1c20-4ada-81fc-6528a6035a69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:09:10.176817 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176726 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fda375d4-1c20-4ada-81fc-6528a6035a69" (UID: "fda375d4-1c20-4ada-81fc-6528a6035a69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:09:10.177015 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176874 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fda375d4-1c20-4ada-81fc-6528a6035a69" (UID: "fda375d4-1c20-4ada-81fc-6528a6035a69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:09:10.177015 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.176985 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:10.177015 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.177005 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-credential-socket\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:10.177015 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.177003 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda375d4-1c20-4ada-81fc-6528a6035a69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fda375d4-1c20-4ada-81fc-6528a6035a69" (UID: "fda375d4-1c20-4ada-81fc-6528a6035a69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:09:10.177251 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.177020 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-workload-certs\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:10.177374 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.177353 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-data" (OuterVolumeSpecName: "istio-data") pod "fda375d4-1c20-4ada-81fc-6528a6035a69" (UID: "fda375d4-1c20-4ada-81fc-6528a6035a69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:09:10.178607 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.178582 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-token" (OuterVolumeSpecName: "istio-token") pod "fda375d4-1c20-4ada-81fc-6528a6035a69" (UID: "fda375d4-1c20-4ada-81fc-6528a6035a69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:09:10.178736 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.178718 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fda375d4-1c20-4ada-81fc-6528a6035a69" (UID: "fda375d4-1c20-4ada-81fc-6528a6035a69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 14:09:10.178891 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.178873 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-kube-api-access-7xddd" (OuterVolumeSpecName: "kube-api-access-7xddd") pod "fda375d4-1c20-4ada-81fc-6528a6035a69" (UID: "fda375d4-1c20-4ada-81fc-6528a6035a69"). InnerVolumeSpecName "kube-api-access-7xddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:09:10.178947 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.178934 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fda375d4-1c20-4ada-81fc-6528a6035a69" (UID: "fda375d4-1c20-4ada-81fc-6528a6035a69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:09:10.230358 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.230336 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tx2mg_e07431ca-19e6-405b-8303-174e7a32d40a/manager/0.log" Jul 07 14:09:10.278133 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.278112 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xddd\" (UniqueName: \"kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-kube-api-access-7xddd\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:10.278133 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.278132 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-token\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:10.278276 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.278141 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-data\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:10.278276 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.278151 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-podinfo\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:10.278276 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.278160 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda375d4-1c20-4ada-81fc-6528a6035a69-istiod-ca-cert\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:10.278276 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.278190 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda375d4-1c20-4ada-81fc-6528a6035a69-istio-envoy\") on node \"ip-10-0-132-236.ec2.internal\" DevicePath \"\"" Jul 07 14:09:10.352623 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.352599 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qmp7x_42e7da0e-84bc-4757-bccc-26d24e0bd443/manager/0.log" Jul 07 14:09:10.456817 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.456738 2576 generic.go:356] "Generic (PLEG): container finished" podID="fda375d4-1c20-4ada-81fc-6528a6035a69" containerID="658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378" exitCode=0 Jul 07 14:09:10.456817 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.456798 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" Jul 07 14:09:10.456817 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.456812 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" event={"ID":"fda375d4-1c20-4ada-81fc-6528a6035a69","Type":"ContainerDied","Data":"658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378"} Jul 07 14:09:10.457029 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.456867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9" event={"ID":"fda375d4-1c20-4ada-81fc-6528a6035a69","Type":"ContainerDied","Data":"8f0452aff58aba92e4a85d0e6b559911aa7333a0c1b18b4c4618ec260c962e6c"} Jul 07 14:09:10.457029 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.456900 2576 scope.go:117] "RemoveContainer" containerID="658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378" Jul 07 14:09:10.466330 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.466314 2576 scope.go:117] "RemoveContainer" containerID="4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932" Jul 07 14:09:10.474279 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.474262 2576 scope.go:117] "RemoveContainer" containerID="658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378" Jul 07 14:09:10.474533 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:09:10.474512 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378\": container with ID starting with 658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378 not found: ID does not exist" containerID="658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378" Jul 07 14:09:10.474609 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.474540 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378"} err="failed to get container status \"658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378\": rpc error: code = NotFound desc = could not find container \"658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378\": container with ID starting with 658f3ca9d9864c9b48dd42e1f402102383df42cd5073a59ff806102cf30ae378 not found: ID does not exist" Jul 07 14:09:10.474609 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.474558 2576 scope.go:117] "RemoveContainer" containerID="4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932" Jul 07 14:09:10.474801 ip-10-0-132-236 kubenswrapper[2576]: E0707 14:09:10.474785 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932\": container with ID starting with 4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932 not found: ID does not exist" containerID="4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932" Jul 07 14:09:10.474887 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.474802 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932"} err="failed to get container status \"4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932\": rpc error: code = NotFound desc = could not find container \"4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932\": container with ID starting with 4264b44a313b3019230c01757fed92e953044fa660c3f6455876ae9845363932 not found: ID does not exist" Jul 07 14:09:10.480431 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.480416 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-ffb787797-6w54p_87cd98f5-c09f-419b-bc9d-1bf69143d222/maas-api/0.log" Jul 07 14:09:10.495807 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.495786 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9"] Jul 07 14:09:10.497635 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.497613 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-f0b1df-openshift-default-d7d968c6c-bw2h9"] Jul 07 14:09:10.590058 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.590037 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fda375d4-1c20-4ada-81fc-6528a6035a69" path="/var/lib/kubelet/pods/fda375d4-1c20-4ada-81fc-6528a6035a69/volumes" Jul 07 14:09:10.594740 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.594723 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723865-zr2j5_ca5f68e4-9118-4fa2-b23a-466d07e29f74/cleanup/0.log" Jul 07 14:09:10.713095 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.713037 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723880-czfzp_e2bef877-d26f-433b-a68a-cf29ae46737a/cleanup/0.log" Jul 07 14:09:10.838575 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.838546 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d798ccfdb-x8k6c_5c4dab78-b0c0-4626-9a73-a0fa4ab64b40/manager/0.log" Jul 07 14:09:10.952995 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:10.952974 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9s5wf_865a955f-a66c-426e-a54b-05dd7cb5c85e/server/0.log" Jul 07 14:09:11.339760 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:11.339732 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76f9489749-nlmcb_3f990869-3802-4dbb-b41b-15e6832bf9a1/copy-manifests/0.log" Jul 07 14:09:11.349106 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:11.349075 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76f9489749-nlmcb_3f990869-3802-4dbb-b41b-15e6832bf9a1/manager/0.log" Jul 07 14:09:11.610056 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:11.609980 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-lcwrd_130dfdad-44a1-4dab-8e2b-01c3d2ecf09c/postgres/0.log" Jul 07 14:09:13.144481 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:13.144450 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bmq67_d10f6a7b-152e-4a79-b1fb-0095c9a2b3c0/manager/0.log" Jul 07 14:09:13.254466 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:13.254442 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nj7zn_e6873865-58c6-4ec2-83d8-6e84cd7df9cc/kuadrant-console-plugin/0.log" Jul 07 14:09:13.370155 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:13.370127 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-w496l_f763fb5c-d2d5-4832-b013-109b98dc92fb/registry-server/0.log" Jul 07 14:09:13.752979 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:13.752953 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-l8bx8_6720d4e8-917a-4f91-a721-085bd04a35fa/manager/0.log" Jul 07 14:09:14.118448 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:14.118380 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4s7csp_031f9d9e-0d19-4198-a277-3c7a8e4d5469/istio-proxy/0.log" Jul 07 14:09:14.357321 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:14.357290 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-87699d5c5-6zj4w_63401851-59e7-4ee7-9df8-73148808810d/kube-auth-proxy/0.log" Jul 07 14:09:14.583433 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:14.583401 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-4qqm6_dcca19d0-38dd-4c17-b2a6-c9520788d994/istio-proxy/1.log" Jul 07 14:09:14.709557 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:14.709529 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-w6qzb_23b5fcf7-a5c9-4bed-9626-d155169c8e09/payload-pre-processing/0.log" Jul 07 14:09:14.828052 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:14.828025 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-tn72h_db1dc435-fd83-456e-82ea-1848a38ac77b/payload-processing/0.log" Jul 07 14:09:15.299079 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:15.299055 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jhbb6_1bd5d6f6-20f9-48d0-9f1b-951d8e25b0fe/main/0.log" Jul 07 14:09:15.417581 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:15.417557 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-r4vt9_6b2ca1d6-24ff-43b1-b612-8ca4930da245/main/0.log" Jul 07 14:09:15.763400 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:15.763374 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ctxlx_b9159cd8-20b8-492b-9a7a-58652ea68caf/main/0.log" Jul 07 14:09:22.518497 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:22.518466 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dcg6d_32f0bc68-c8ce-4756-8944-97f48df4e7cb/global-pull-secret-syncer/0.log" Jul 07 14:09:22.726789 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:22.726756 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ns4bt_b4edb049-06f0-4602-9c6f-592e4d35ca14/konnectivity-agent/0.log" Jul 07 14:09:22.777713 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:22.777639 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-236.ec2.internal_18bcd73158ca1a5b1c82fc4d26d8e309/haproxy/0.log" Jul 07 14:09:27.199943 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:27.199912 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bmq67_d10f6a7b-152e-4a79-b1fb-0095c9a2b3c0/manager/0.log" Jul 07 14:09:27.234730 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:27.234709 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nj7zn_e6873865-58c6-4ec2-83d8-6e84cd7df9cc/kuadrant-console-plugin/0.log" Jul 07 14:09:27.284205 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:27.284161 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-w496l_f763fb5c-d2d5-4832-b013-109b98dc92fb/registry-server/0.log" Jul 07 14:09:27.761413 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:27.761377 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-l8bx8_6720d4e8-917a-4f91-a721-085bd04a35fa/manager/0.log" Jul 07 14:09:30.239519 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:30.239488 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkq9g_f41a94cc-0686-41ca-9666-2223a9736bd7/node-exporter/0.log" Jul 07 14:09:30.261926 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:30.261901 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkq9g_f41a94cc-0686-41ca-9666-2223a9736bd7/kube-rbac-proxy/0.log" Jul 07 14:09:30.287253 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:30.287231 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkq9g_f41a94cc-0686-41ca-9666-2223a9736bd7/init-textfile/0.log" Jul 07 14:09:32.346198 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:32.346154 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-qd2sw_0f2851ac-4bb5-4cb4-957b-146cec3af69c/networking-console-plugin/0.log" Jul 07 14:09:33.306179 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306140 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp"] Jul 07 14:09:33.306516 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306502 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" Jul 07 14:09:33.306565 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306518 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" Jul 07 14:09:33.306565 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306530 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" Jul 07 14:09:33.306565 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306536 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77aa0b3b-0f2c-48f2-972c-2a95cfde92fa" containerName="istio-proxy" Jul 07 14:09:33.306565 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306541 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda375d4-1c20-4ada-81fc-6528a6035a69" containerName="istio-proxy" Jul 07 14:09:33.306565 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306547 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda375d4-1c20-4ada-81fc-6528a6035a69" containerName="istio-proxy" Jul 07 14:09:33.306565 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306558 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" Jul 07 14:09:33.306565 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306563 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" Jul 07 14:09:33.306799 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306631 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda375d4-1c20-4ada-81fc-6528a6035a69" containerName="istio-proxy" Jul 07 14:09:33.306799 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306639 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" Jul 07 14:09:33.306799 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.306645 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f93c2b-9cc2-45f2-978b-9795e63423ac" containerName="istio-proxy" Jul 07 14:09:33.309098 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.309080 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.312276 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.312256 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kxg6n\"/\"kube-root-ca.crt\"" Jul 07 14:09:33.313418 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.313400 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kxg6n\"/\"default-dockercfg-nkdqx\"" Jul 07 14:09:33.313514 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.313402 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kxg6n\"/\"openshift-service-ca.crt\"" Jul 07 14:09:33.318276 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.318254 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp"] Jul 07 14:09:33.367049 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.367030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-lib-modules\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.367380 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.367062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-proc\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.367380 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.367103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-podres\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.367380 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.367138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-sys\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.367380 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.367235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzvn\" (UniqueName: \"kubernetes.io/projected/21835337-43ee-4119-a8ee-2ecb98ddd7dd-kube-api-access-lzzvn\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.468376 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.468347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-podres\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.468473 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.468383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-sys\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.468473 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.468436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzvn\" (UniqueName: \"kubernetes.io/projected/21835337-43ee-4119-a8ee-2ecb98ddd7dd-kube-api-access-lzzvn\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.468545 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.468471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-lib-modules\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.468545 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.468501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-proc\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.468545 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.468529 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-podres\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.468643 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.468544 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-sys\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.468643 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.468593 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-proc\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.468643 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.468618 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21835337-43ee-4119-a8ee-2ecb98ddd7dd-lib-modules\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.478519 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.478500 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzvn\" (UniqueName: \"kubernetes.io/projected/21835337-43ee-4119-a8ee-2ecb98ddd7dd-kube-api-access-lzzvn\") pod \"perf-node-gather-daemonset-lgdsp\" (UID: \"21835337-43ee-4119-a8ee-2ecb98ddd7dd\") " pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.568176 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.568103 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-zzfs7_ac8db3f2-4183-4674-95b3-b7ea6ab06b58/download-server/0.log" Jul 07 14:09:33.619218 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.619196 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:33.749593 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:33.749564 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp"] Jul 07 14:09:33.750701 ip-10-0-132-236 kubenswrapper[2576]: W0707 14:09:33.750676 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod21835337_43ee_4119_a8ee_2ecb98ddd7dd.slice/crio-b36d876ec891815f62301dd7462f4226afe9fecacc6874e2e9bd39fde29bc576 WatchSource:0}: Error finding container b36d876ec891815f62301dd7462f4226afe9fecacc6874e2e9bd39fde29bc576: Status 404 returned error can't find the container with id b36d876ec891815f62301dd7462f4226afe9fecacc6874e2e9bd39fde29bc576 Jul 07 14:09:34.574060 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:34.574022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" event={"ID":"21835337-43ee-4119-a8ee-2ecb98ddd7dd","Type":"ContainerStarted","Data":"8feac69fa98612eae0e4723e80c38efded469bfcce37ab248ada69abb4169fdd"} Jul 07 14:09:34.574484 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:34.574081 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" event={"ID":"21835337-43ee-4119-a8ee-2ecb98ddd7dd","Type":"ContainerStarted","Data":"b36d876ec891815f62301dd7462f4226afe9fecacc6874e2e9bd39fde29bc576"} Jul 07 14:09:34.574484 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:34.574187 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:34.591867 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:34.591819 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" podStartSLOduration=1.591803773 podStartE2EDuration="1.591803773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:34.591104975 +0000 UTC m=+2508.592346503" watchObservedRunningTime="2026-07-07 14:09:34.591803773 +0000 UTC m=+2508.593045299" Jul 07 14:09:35.331125 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:35.331100 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xbmhz_5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1/dns/0.log" Jul 07 14:09:35.371343 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:35.371319 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xbmhz_5d8603c4-97f0-499b-8cc4-ae1ea0b71ea1/kube-rbac-proxy/0.log" Jul 07 14:09:35.477084 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:35.477053 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qhhwm_0165e63b-34ba-494b-9a7b-31eb2e652b54/dns-node-resolver/0.log" Jul 07 14:09:36.025106 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:36.025075 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84dcfb865b-s2qps_b2bd6ca3-d88a-456f-913c-0ba058905a66/registry/0.log" Jul 07 14:09:36.073538 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:36.073513 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wcqgp_17a9a9ab-5b3b-4805-b971-3be2d30b46ec/node-ca/0.log" Jul 07 14:09:37.067045 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:37.067012 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4s7csp_031f9d9e-0d19-4198-a277-3c7a8e4d5469/istio-proxy/0.log" Jul 07 14:09:37.206902 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:37.206873 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-87699d5c5-6zj4w_63401851-59e7-4ee7-9df8-73148808810d/kube-auth-proxy/0.log" Jul 07 14:09:37.267669 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:37.267641 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-4qqm6_dcca19d0-38dd-4c17-b2a6-c9520788d994/istio-proxy/0.log" Jul 07 14:09:37.292428 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:37.292401 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-4qqm6_dcca19d0-38dd-4c17-b2a6-c9520788d994/istio-proxy/1.log" Jul 07 14:09:37.321567 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:37.321502 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-w6qzb_23b5fcf7-a5c9-4bed-9626-d155169c8e09/payload-pre-processing/0.log" Jul 07 14:09:37.372875 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:37.372848 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-tn72h_db1dc435-fd83-456e-82ea-1848a38ac77b/payload-processing/0.log" Jul 07 14:09:38.156995 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:38.156966 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hzkh9_3b7cf248-7634-4651-b3d0-cb6f11b5bede/serve-healthcheck-canary/0.log" Jul 07 14:09:38.760896 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:38.760869 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rz9rq_4c603960-bc78-4176-bcce-153840f1df0b/kube-rbac-proxy/0.log" Jul 07 14:09:38.785588 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:38.785557 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rz9rq_4c603960-bc78-4176-bcce-153840f1df0b/exporter/0.log" Jul 07 14:09:38.807139 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:38.807114 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rz9rq_4c603960-bc78-4176-bcce-153840f1df0b/extractor/0.log" Jul 07 14:09:40.589604 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:40.589578 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kxg6n/perf-node-gather-daemonset-lgdsp" Jul 07 14:09:41.046672 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:41.046638 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tx2mg_e07431ca-19e6-405b-8303-174e7a32d40a/manager/0.log" Jul 07 14:09:41.172789 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:41.172756 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qmp7x_42e7da0e-84bc-4757-bccc-26d24e0bd443/manager/0.log" Jul 07 14:09:41.212049 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:41.212021 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-ffb787797-6w54p_87cd98f5-c09f-419b-bc9d-1bf69143d222/maas-api/0.log" Jul 07 14:09:41.238782 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:41.238752 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723865-zr2j5_ca5f68e4-9118-4fa2-b23a-466d07e29f74/cleanup/0.log" Jul 07 14:09:41.272073 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:41.272049 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723880-czfzp_e2bef877-d26f-433b-a68a-cf29ae46737a/cleanup/0.log" Jul 07 14:09:41.377158 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:41.377077 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d798ccfdb-x8k6c_5c4dab78-b0c0-4626-9a73-a0fa4ab64b40/manager/0.log" Jul 07 14:09:41.402703 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:41.402675 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9s5wf_865a955f-a66c-426e-a54b-05dd7cb5c85e/server/0.log" Jul 07 14:09:41.579641 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:41.579611 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76f9489749-nlmcb_3f990869-3802-4dbb-b41b-15e6832bf9a1/manager/0.log" Jul 07 14:09:41.610139 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:41.610112 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76f9489749-nlmcb_3f990869-3802-4dbb-b41b-15e6832bf9a1/copy-manifests/0.log" Jul 07 14:09:41.744245 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:41.744218 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-lcwrd_130dfdad-44a1-4dab-8e2b-01c3d2ecf09c/postgres/0.log" Jul 07 14:09:42.848490 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:42.848461 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4s7csp_031f9d9e-0d19-4198-a277-3c7a8e4d5469/istio-proxy/0.log" Jul 07 14:09:42.970127 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:42.970094 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-87699d5c5-6zj4w_63401851-59e7-4ee7-9df8-73148808810d/kube-auth-proxy/0.log" Jul 07 14:09:43.012497 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:43.012466 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-4qqm6_dcca19d0-38dd-4c17-b2a6-c9520788d994/istio-proxy/0.log" Jul 07 14:09:43.039082 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:43.039054 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-4qqm6_dcca19d0-38dd-4c17-b2a6-c9520788d994/istio-proxy/1.log" Jul 07 14:09:43.059175 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:43.059139 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-w6qzb_23b5fcf7-a5c9-4bed-9626-d155169c8e09/payload-pre-processing/0.log" Jul 07 14:09:43.102460 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:43.102428 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-tn72h_db1dc435-fd83-456e-82ea-1848a38ac77b/payload-processing/0.log" Jul 07 14:09:43.399958 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:43.399933 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9h9d9_dcc67eb5-eeea-4f74-b49a-7ced64b1be44/openshift-lws-operator/0.log" Jul 07 14:09:44.344080 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:44.344037 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4s7csp_031f9d9e-0d19-4198-a277-3c7a8e4d5469/istio-proxy/0.log" Jul 07 14:09:44.475059 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:44.475032 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-87699d5c5-6zj4w_63401851-59e7-4ee7-9df8-73148808810d/kube-auth-proxy/0.log" Jul 07 14:09:44.515271 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:44.515222 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-4qqm6_dcca19d0-38dd-4c17-b2a6-c9520788d994/istio-proxy/0.log" Jul 07 14:09:44.543350 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:44.543325 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-4qqm6_dcca19d0-38dd-4c17-b2a6-c9520788d994/istio-proxy/1.log" Jul 07 14:09:44.561862 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:44.561839 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-w6qzb_23b5fcf7-a5c9-4bed-9626-d155169c8e09/payload-pre-processing/0.log" Jul 07 14:09:44.601039 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:44.601016 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-tn72h_db1dc435-fd83-456e-82ea-1848a38ac77b/payload-processing/0.log" Jul 07 14:09:48.632201 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:48.632161 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-tsxzc_f2d1f3d3-c447-4f28-ab19-c9ae8554f58b/migrator/0.log" Jul 07 14:09:48.654767 ip-10-0-132-236 kubenswrapper[2576]: I0707 14:09:48.654741 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-tsxzc_f2d1f3d3-c447-4f28-ab19-c9ae8554f58b/graceful-termination/0.log"