Sep 07 03:46:18.960421 ip-10-0-131-215 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 07 03:46:18.960432 ip-10-0-131-215 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 07 03:46:18.960439 ip-10-0-131-215 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 07 03:46:18.960674 ip-10-0-131-215 systemd[1]: Failed to start Kubernetes Kubelet. Sep 07 03:46:30.434676 ip-10-0-131-215 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 07 03:46:30.434691 ip-10-0-131-215 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e0acb95f8d5946a5b75fb89ccfe2158a -- Sep 07 03:48:01.313325 ip-10-0-131-215 systemd[1]: Starting Kubernetes Kubelet... Sep 07 03:48:01.751078 ip-10-0-131-215 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 03:48:01.751078 ip-10-0-131-215 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 07 03:48:01.751078 ip-10-0-131-215 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 03:48:01.751078 ip-10-0-131-215 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 07 03:48:01.751078 ip-10-0-131-215 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 03:48:01.753192 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.753108 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 07 03:48:01.758449 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758426 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 03:48:01.758449 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758445 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 03:48:01.758449 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758449 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 03:48:01.758449 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758452 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 03:48:01.758449 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758455 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758458 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758462 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758465 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758468 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758471 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758473 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758476 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758479 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758482 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758484 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758487 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758490 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758493 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758497 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758500 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758503 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758506 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758508 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 03:48:01.758644 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758511 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758513 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758517 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758521 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758527 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758530 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758533 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758535 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758538 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758541 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758543 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758546 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758548 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758551 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758554 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758557 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758560 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758563 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758566 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 03:48:01.759114 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758568 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758571 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758574 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758576 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758579 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758582 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758584 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758587 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758590 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758592 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758595 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758598 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758601 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758604 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758606 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758609 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758614 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758617 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758620 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 03:48:01.759599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758623 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758626 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758628 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758631 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758634 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758636 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758639 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758642 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758644 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758649 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758652 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758655 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758659 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758662 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758664 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758667 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758670 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758672 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758675 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758677 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 03:48:01.760176 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758680 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758682 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758685 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758688 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758691 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758694 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758696 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758699 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758701 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758704 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758706 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758710 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758712 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758716 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758719 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758721 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758724 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.758727 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759424 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759431 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 03:48:01.760665 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759434 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759436 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759439 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759442 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759444 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759447 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759450 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759453 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759455 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759458 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759461 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759465 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759468 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759470 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759473 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759485 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759487 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759490 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759493 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 03:48:01.761163 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759497 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759501 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759504 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759507 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759510 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759513 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759517 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759520 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759523 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759526 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759529 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759532 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759535 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759537 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759541 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759545 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759547 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759550 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759553 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 03:48:01.761635 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759555 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759558 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759560 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759564 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759566 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759569 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759571 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759573 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759576 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759578 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759581 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759584 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759587 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759590 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759592 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759595 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759598 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759601 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759603 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759607 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 03:48:01.762146 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759609 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759612 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759615 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759617 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759620 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759623 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759626 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759630 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759632 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759635 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759637 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759640 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759642 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759645 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759647 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759650 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759653 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759655 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759658 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 03:48:01.762636 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759660 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759662 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759665 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759667 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759670 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759673 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759676 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759678 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759681 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759684 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759687 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759690 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759692 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759695 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759698 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759700 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759703 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759705 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759708 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.759710 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 03:48:01.763104 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760105 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760110 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760113 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760116 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760119 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760122 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760125 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760128 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760130 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760133 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760136 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760138 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760141 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760143 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760146 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760148 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760151 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760154 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760156 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760159 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 03:48:01.763595 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760162 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760164 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760167 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760171 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760179 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760182 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760185 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760188 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760190 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760193 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760196 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760198 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760201 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760203 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760206 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760209 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760211 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760214 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760217 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 07 03:48:01.764075 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760219 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760223 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760227 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760231 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760234 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760237 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760239 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760242 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760246 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760248 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760252 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760254 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760257 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760260 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760263 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760265 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760267 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760271 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760273 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760276 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 03:48:01.764541 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760278 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760280 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760283 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760285 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760288 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760290 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760293 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760295 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760298 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760300 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760303 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760306 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760308 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760311 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760313 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760316 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760319 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760322 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760325 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760327 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 03:48:01.765016 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760330 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760332 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760335 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760338 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760340 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760342 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760345 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760347 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760349 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760352 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760355 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760358 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760360 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760363 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760365 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760368 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760371 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760373 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760375 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.760378 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 03:48:01.765513 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760455 2555 flags.go:64] FLAG: --address="0.0.0.0" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760491 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760499 2555 flags.go:64] FLAG: --anonymous-auth="true" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760503 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760508 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760511 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760515 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760519 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760522 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760525 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760529 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760532 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760535 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760538 2555 flags.go:64] FLAG: --cgroup-root="" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760541 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760544 2555 flags.go:64] FLAG: --client-ca-file="" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760547 2555 flags.go:64] FLAG: --cloud-provider="external" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760550 2555 flags.go:64] FLAG: --cluster-dns="[]" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760554 2555 flags.go:64] FLAG: --cluster-domain="" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760557 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760560 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760563 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760567 2555 flags.go:64] FLAG: --container-log-max-files="5" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760570 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 07 03:48:01.765992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760573 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760576 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760580 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760582 2555 flags.go:64] FLAG: --contention-profiling="false" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760585 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760588 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760591 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760595 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760599 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760602 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760607 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760609 2555 flags.go:64] FLAG: --enable-load-reader="false" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760613 2555 flags.go:64] FLAG: --enable-server="true" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760616 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760621 2555 flags.go:64] FLAG: --event-burst="100" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760624 2555 flags.go:64] FLAG: --event-qps="50" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760627 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760630 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760633 2555 flags.go:64] FLAG: --eviction-hard="" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760636 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760639 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760642 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760645 2555 flags.go:64] FLAG: --eviction-soft="" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760648 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760650 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 07 03:48:01.766576 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760654 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760657 2555 flags.go:64] FLAG: --experimental-mounter-path="" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760660 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760663 2555 flags.go:64] FLAG: --fail-swap-on="true" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760665 2555 flags.go:64] FLAG: --feature-gates="" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760670 2555 flags.go:64] FLAG: --file-check-frequency="20s" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760673 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760676 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760679 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760682 2555 flags.go:64] FLAG: --healthz-port="10248" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760685 2555 flags.go:64] FLAG: --help="false" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760688 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-131-215.ec2.internal" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760691 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760694 2555 flags.go:64] FLAG: --http-check-frequency="20s" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760697 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760700 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760703 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760708 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760711 2555 flags.go:64] FLAG: --image-service-endpoint="" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760713 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760716 2555 flags.go:64] FLAG: --kube-api-burst="100" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760719 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760722 2555 flags.go:64] FLAG: --kube-api-qps="50" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760725 2555 flags.go:64] FLAG: --kube-reserved="" Sep 07 03:48:01.767214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760728 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760731 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760733 2555 flags.go:64] FLAG: --kubelet-cgroups="" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760737 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760739 2555 flags.go:64] FLAG: --lock-file="" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760742 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760745 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760748 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760754 2555 flags.go:64] FLAG: --log-json-split-stream="false" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760757 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760760 2555 flags.go:64] FLAG: --log-text-split-stream="false" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760763 2555 flags.go:64] FLAG: --logging-format="text" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760767 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760770 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760773 2555 flags.go:64] FLAG: --manifest-url="" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760775 2555 flags.go:64] FLAG: --manifest-url-header="" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760780 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760783 2555 flags.go:64] FLAG: --max-open-files="1000000" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760787 2555 flags.go:64] FLAG: --max-pods="110" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760789 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760792 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760795 2555 flags.go:64] FLAG: --memory-manager-policy="None" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760798 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760801 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760804 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 07 03:48:01.767852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760808 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760815 2555 flags.go:64] FLAG: --node-status-max-images="50" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760818 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760821 2555 flags.go:64] FLAG: --oom-score-adj="-999" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760824 2555 flags.go:64] FLAG: --pod-cidr="" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760827 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760833 2555 flags.go:64] FLAG: --pod-manifest-path="" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760835 2555 flags.go:64] FLAG: --pod-max-pids="-1" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760839 2555 flags.go:64] FLAG: --pods-per-core="0" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760841 2555 flags.go:64] FLAG: --port="10250" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760845 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760848 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ed9b29accd629a4b" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760851 2555 flags.go:64] FLAG: --qos-reserved="" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760854 2555 flags.go:64] FLAG: --read-only-port="10255" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760857 2555 flags.go:64] FLAG: --register-node="true" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760859 2555 flags.go:64] FLAG: --register-with-taints="" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760863 2555 flags.go:64] FLAG: --registry-burst="10" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760867 2555 flags.go:64] FLAG: --registry-qps="5" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760870 2555 flags.go:64] FLAG: --reserved-cpus="" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760872 2555 flags.go:64] FLAG: --reserved-memory="" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760877 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760880 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760883 2555 flags.go:64] FLAG: --rotate-certificates="false" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760886 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760889 2555 flags.go:64] FLAG: --runonce="false" Sep 07 03:48:01.768469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760892 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760895 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760898 2555 flags.go:64] FLAG: --seccomp-default="false" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760901 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760903 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760906 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760909 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760913 2555 flags.go:64] FLAG: --storage-driver-password="root" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760916 2555 flags.go:64] FLAG: --storage-driver-secure="false" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760919 2555 flags.go:64] FLAG: --storage-driver-table="stats" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760922 2555 flags.go:64] FLAG: --storage-driver-user="root" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760925 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760928 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760931 2555 flags.go:64] FLAG: --system-cgroups="" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760934 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760939 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760942 2555 flags.go:64] FLAG: --tls-cert-file="" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760944 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760949 2555 flags.go:64] FLAG: --tls-min-version="" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760952 2555 flags.go:64] FLAG: --tls-private-key-file="" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760954 2555 flags.go:64] FLAG: --topology-manager-policy="none" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760957 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760960 2555 flags.go:64] FLAG: --topology-manager-scope="container" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760963 2555 flags.go:64] FLAG: --v="2" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760967 2555 flags.go:64] FLAG: --version="false" Sep 07 03:48:01.769091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760972 2555 flags.go:64] FLAG: --vmodule="" Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760975 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.760980 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761110 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761114 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761117 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761120 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761123 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761127 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761129 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761132 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761135 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761137 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761141 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761145 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761148 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761150 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761153 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761155 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761158 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761160 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 03:48:01.769702 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761163 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761165 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761168 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761171 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761175 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761179 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761181 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761184 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761186 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761189 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761191 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761194 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761198 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761203 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761206 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761208 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761211 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761214 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761217 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 03:48:01.770230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761219 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761222 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761224 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761227 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761230 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761232 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761237 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761239 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761242 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761244 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761247 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761250 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761252 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761255 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761258 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761260 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761263 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761265 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761269 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 03:48:01.770741 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761272 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761275 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761278 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761281 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761284 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761287 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761289 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761294 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761296 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761299 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761301 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761304 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761306 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761309 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761312 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761315 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761317 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761320 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761323 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761326 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 03:48:01.771222 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761329 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761331 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761334 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761336 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761339 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761341 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761344 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761346 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761349 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761351 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761355 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761358 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761362 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761366 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761370 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761373 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761375 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761378 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761380 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761386 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 03:48:01.771722 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761390 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761393 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.761396 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.762075 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.769809 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.769825 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.770772 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.770783 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770844 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770849 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770852 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770855 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770859 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770862 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770865 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 03:48:01.772230 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770867 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770870 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770873 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770876 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770879 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770882 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770885 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770887 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770890 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770892 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770895 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770897 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770900 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770902 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770905 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770909 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770913 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770918 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770921 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 03:48:01.772615 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770923 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770926 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770929 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770932 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770935 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770938 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770940 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770943 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770946 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770948 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770951 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770953 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770956 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770959 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770961 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770964 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770966 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770969 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770971 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770974 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 03:48:01.773082 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770977 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770979 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770981 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770984 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770987 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770989 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770991 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770994 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770996 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.770999 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771002 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771005 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771008 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771012 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771016 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771020 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771023 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771026 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771029 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 03:48:01.773576 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771032 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771034 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771037 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771040 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771043 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771045 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771048 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771051 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771067 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771070 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771073 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771076 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771078 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771081 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771084 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771086 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771089 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771092 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771094 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771098 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 03:48:01.774039 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771100 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771103 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771106 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771108 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771110 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771114 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771116 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771119 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771122 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771125 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771127 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771130 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771132 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771135 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 03:48:01.774571 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.771140 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771250 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771255 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771258 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771261 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771264 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771266 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771269 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771271 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771274 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771276 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771279 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771281 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771284 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771286 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771289 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771291 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771294 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771296 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771299 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 03:48:01.774916 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771301 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771304 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771306 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771309 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771312 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771315 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771317 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771320 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771322 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771325 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771327 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771330 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771332 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771335 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771337 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771340 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771342 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771345 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771347 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771350 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 03:48:01.775413 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771352 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771355 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771358 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771360 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771362 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771365 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771369 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771372 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771375 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771378 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771380 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771383 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771386 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771390 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771393 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771396 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771399 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771402 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 03:48:01.775890 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771405 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771408 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771410 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771413 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771415 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771418 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771421 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771424 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771427 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771429 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771432 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771434 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771437 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771439 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771441 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771444 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771446 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771449 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771451 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771454 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 03:48:01.776330 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771456 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771460 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771462 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771465 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771467 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771470 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771473 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771482 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771485 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771488 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771490 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771494 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771498 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771501 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771503 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771506 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771509 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771511 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771514 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771516 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 03:48:01.776804 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771518 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 03:48:01.777291 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:01.771521 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 03:48:01.777291 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.771526 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 03:48:01.777291 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.771621 2555 server.go:962] "Client rotation is on, will bootstrap in background" Sep 07 03:48:01.777291 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.773959 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 07 03:48:01.777291 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.775029 2555 server.go:1019] "Starting client certificate rotation" Sep 07 03:48:01.777291 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.775087 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 03:48:01.777291 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.775120 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 03:48:01.777810 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.777797 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 03:48:01.782235 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.782211 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 03:48:01.792882 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.792865 2555 log.go:25] "Validated CRI v1 runtime API" Sep 07 03:48:01.796464 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.796447 2555 log.go:25] "Validated CRI v1 image API" Sep 07 03:48:01.797336 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.797314 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 07 03:48:01.802727 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.802700 2555 fs.go:135] Filesystem UUIDs: map[72304a64-4f01-4d33-b162-890e4e9d38d1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 855d0631-2e16-4562-8261-07b06c9e55cc:/dev/nvme0n1p4] Sep 07 03:48:01.802801 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.802725 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 07 03:48:01.805743 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.805727 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 03:48:01.808027 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.807917 2555 manager.go:216] Machine: {Timestamp:2026-09-07 03:48:01.806351457 +0000 UTC m=+0.381217737 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099809 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a1472d0d1918ce48c5123693e8393 SystemUUID:ec2a1472-d0d1-918c-e48c-5123693e8393 BootID:e0acb95f-8d59-46a5-b75f-b89ccfe2158a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1f:da:e7:40:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1f:da:e7:40:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:69:39:ee:b3:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 07 03:48:01.808027 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.808023 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 07 03:48:01.808188 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.808108 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 07 03:48:01.811502 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.811482 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 07 03:48:01.811644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.811505 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-215.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 07 03:48:01.812383 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.812373 2555 topology_manager.go:138] "Creating topology manager with none policy" Sep 07 03:48:01.812419 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.812385 2555 container_manager_linux.go:309] "Creating device plugin manager" Sep 07 03:48:01.812419 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.812393 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 03:48:01.812419 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.812406 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 03:48:01.814449 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.814440 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 07 03:48:01.818933 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.818923 2555 state_mem.go:36] "Initialized new in-memory state store" Sep 07 03:48:01.819051 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.819042 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 07 03:48:01.819107 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.819081 2555 kubelet.go:486] "Attempting to sync node with API server" Sep 07 03:48:01.819107 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.819091 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 07 03:48:01.819107 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.819106 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 07 03:48:01.819234 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.819116 2555 kubelet.go:398] "Adding apiserver pod source" Sep 07 03:48:01.819234 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.819125 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 07 03:48:01.819494 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.819482 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 03:48:01.819533 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.819505 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 03:48:01.821895 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.821877 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 07 03:48:01.824794 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.824780 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 07 03:48:01.824865 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.824808 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 07 03:48:01.825588 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825576 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 07 03:48:01.825633 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825593 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 07 03:48:01.825633 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825599 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 07 03:48:01.825633 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825604 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 07 03:48:01.825633 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825613 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 07 03:48:01.825633 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825618 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 07 03:48:01.825762 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825638 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 07 03:48:01.825762 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825644 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 07 03:48:01.825762 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825650 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 07 03:48:01.825762 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825655 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 07 03:48:01.825762 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825666 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 07 03:48:01.825762 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.825671 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 07 03:48:01.826501 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.826491 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 07 03:48:01.826501 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.826500 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 07 03:48:01.829726 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.829677 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-215.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 07 03:48:01.829819 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.829730 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 07 03:48:01.829819 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.829759 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-215.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 07 03:48:01.830790 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.830763 2555 server.go:1268] "Started kubelet" Sep 07 03:48:01.830920 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.830871 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 07 03:48:01.830985 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.830945 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 07 03:48:01.831720 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.831266 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 07 03:48:01.831633 ip-10-0-131-215 systemd[1]: Started Kubernetes Kubelet. Sep 07 03:48:01.832705 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.832684 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 07 03:48:01.834090 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.833944 2555 server.go:310] "Adding debug handlers to kubelet server" Sep 07 03:48:01.838573 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.838555 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 07 03:48:01.839136 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.839117 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 07 03:48:01.839220 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.839167 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 07 03:48:01.839271 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.839261 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:01.839937 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.839919 2555 factory.go:55] Registering systemd factory Sep 07 03:48:01.840022 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.839941 2555 factory.go:223] Registration of the systemd container factory successfully Sep 07 03:48:01.840022 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.839992 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 07 03:48:01.840022 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.840000 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 07 03:48:01.840022 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.840014 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 07 03:48:01.840211 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.840117 2555 reconstruct.go:97] "Volume reconstruction finished" Sep 07 03:48:01.840211 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.840126 2555 reconciler.go:29] "Reconciler: start to sync state" Sep 07 03:48:01.840305 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.840225 2555 factory.go:153] Registering CRI-O factory Sep 07 03:48:01.840305 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.840240 2555 factory.go:223] Registration of the crio container factory successfully Sep 07 03:48:01.840305 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.840286 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 07 03:48:01.840305 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.840302 2555 factory.go:103] Registering Raw factory Sep 07 03:48:01.840487 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.840316 2555 manager.go:1196] Started watching for new ooms in manager Sep 07 03:48:01.840658 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.840645 2555 manager.go:349] Starting recovery of all containers Sep 07 03:48:01.842266 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.841022 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-215.ec2.internal.18d2ed44d07a0bea default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-215.ec2.internal,UID:ip-10-0-131-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-215.ec2.internal,},FirstTimestamp:2026-09-07 03:48:01.830726634 +0000 UTC m=+0.405592915,LastTimestamp:2026-09-07 03:48:01.830726634 +0000 UTC m=+0.405592915,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-215.ec2.internal,}" Sep 07 03:48:01.848758 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.848738 2555 manager.go:354] Recovery completed Sep 07 03:48:01.852779 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.852757 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 07 03:48:01.852844 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.852823 2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-215.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 07 03:48:01.854076 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.854048 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 03:48:01.855867 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.855779 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientMemory" Sep 07 03:48:01.855867 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.855819 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 03:48:01.855867 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.855836 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientPID" Sep 07 03:48:01.856911 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.856896 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 07 03:48:01.856911 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.856904 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 03:48:01.856911 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.856910 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 07 03:48:01.857122 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.856927 2555 state_mem.go:36] "Initialized new in-memory state store" Sep 07 03:48:01.857658 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.857633 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientMemory" Sep 07 03:48:01.857658 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.857658 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 03:48:01.857778 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.857672 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientPID" Sep 07 03:48:01.859020 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.859006 2555 policy_none.go:49] "None policy: Start" Sep 07 03:48:01.859086 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.859024 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 07 03:48:01.859086 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.859033 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 07 03:48:01.859369 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.859353 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l2frc" Sep 07 03:48:01.861041 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.861029 2555 policy_none.go:47] "Start" Sep 07 03:48:01.865130 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.865044 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-215.ec2.internal.18d2ed44d1f8b822 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-215.ec2.internal,UID:ip-10-0-131-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-215.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-215.ec2.internal,},FirstTimestamp:2026-09-07 03:48:01.855805474 +0000 UTC m=+0.430671758,LastTimestamp:2026-09-07 03:48:01.855805474 +0000 UTC m=+0.430671758,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-215.ec2.internal,}" Sep 07 03:48:01.866945 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.866889 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-215.ec2.internal.18d2ed44d1f90af0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-215.ec2.internal,UID:ip-10-0-131-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-215.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-215.ec2.internal,},FirstTimestamp:2026-09-07 03:48:01.855826672 +0000 UTC m=+0.430692954,LastTimestamp:2026-09-07 03:48:01.855826672 +0000 UTC m=+0.430692954,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-215.ec2.internal,}" Sep 07 03:48:01.867373 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.867357 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l2frc" Sep 07 03:48:01.887630 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.887616 2555 manager.go:337] "Starting Device Plugin manager" Sep 07 03:48:01.912946 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.887655 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 07 03:48:01.912946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.887665 2555 server.go:87] "Starting device plugin registration server" Sep 07 03:48:01.912946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.887931 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 07 03:48:01.912946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.887941 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 07 03:48:01.912946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.888036 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 07 03:48:01.912946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.888135 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 07 03:48:01.912946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.888144 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 07 03:48:01.912946 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.888531 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 07 03:48:01.912946 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.888570 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:01.928478 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.928452 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 07 03:48:01.929804 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.929790 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 07 03:48:01.929870 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.929840 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 07 03:48:01.929870 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.929862 2555 kubelet.go:2448] "Starting kubelet main sync loop" Sep 07 03:48:01.929963 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:01.929902 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 07 03:48:01.932540 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.932524 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 03:48:01.988716 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.988700 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 03:48:01.989579 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.989563 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientMemory" Sep 07 03:48:01.989651 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.989592 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 03:48:01.989651 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.989602 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientPID" Sep 07 03:48:01.989651 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:01.989624 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.005774 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.005694 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.005774 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.005718 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-215.ec2.internal\": node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:02.030830 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.030811 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal"] Sep 07 03:48:02.030928 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.030880 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 03:48:02.032142 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.032127 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientMemory" Sep 07 03:48:02.032215 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.032155 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 03:48:02.032215 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.032166 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientPID" Sep 07 03:48:02.034518 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.034508 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 03:48:02.034648 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.034631 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.034695 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.034663 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 03:48:02.035221 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.035200 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientMemory" Sep 07 03:48:02.035289 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.035224 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 03:48:02.035289 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.035238 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientPID" Sep 07 03:48:02.035289 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.035264 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientMemory" Sep 07 03:48:02.035289 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.035289 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 03:48:02.035407 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.035300 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientPID" Sep 07 03:48:02.037612 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.037599 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.037674 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.037623 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 03:48:02.038238 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.038221 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientMemory" Sep 07 03:48:02.038305 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.038252 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 03:48:02.038305 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.038269 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientPID" Sep 07 03:48:02.040666 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.040652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed84f1ba21f69b974e5fd2991bf7d19-config\") pod \"kube-apiserver-proxy-ip-10-0-131-215.ec2.internal\" (UID: \"bed84f1ba21f69b974e5fd2991bf7d19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.040737 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.040673 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e62584f24accf83230cac126e1538660-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal\" (UID: \"e62584f24accf83230cac126e1538660\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.040737 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.040690 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e62584f24accf83230cac126e1538660-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal\" (UID: \"e62584f24accf83230cac126e1538660\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.054567 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.054549 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-215.ec2.internal\" not found" node="ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.058813 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.058796 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-215.ec2.internal\" not found" node="ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.075264 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.075246 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:02.141201 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.141176 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e62584f24accf83230cac126e1538660-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal\" (UID: \"e62584f24accf83230cac126e1538660\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.141201 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.141201 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e62584f24accf83230cac126e1538660-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal\" (UID: \"e62584f24accf83230cac126e1538660\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.141348 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.141215 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed84f1ba21f69b974e5fd2991bf7d19-config\") pod \"kube-apiserver-proxy-ip-10-0-131-215.ec2.internal\" (UID: \"bed84f1ba21f69b974e5fd2991bf7d19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.141348 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.141270 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed84f1ba21f69b974e5fd2991bf7d19-config\") pod \"kube-apiserver-proxy-ip-10-0-131-215.ec2.internal\" (UID: \"bed84f1ba21f69b974e5fd2991bf7d19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.141348 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.141276 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e62584f24accf83230cac126e1538660-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal\" (UID: \"e62584f24accf83230cac126e1538660\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.141348 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.141289 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e62584f24accf83230cac126e1538660-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal\" (UID: \"e62584f24accf83230cac126e1538660\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.175748 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.175724 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:02.276810 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.276734 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:02.357425 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.357405 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.361534 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.361517 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" Sep 07 03:48:02.377177 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.377162 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:02.477699 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.477656 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:02.579435 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.579390 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:02.679926 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.679879 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:02.776185 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.776152 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 07 03:48:02.776671 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.776329 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 07 03:48:02.780813 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.780792 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:02.870281 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.870199 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 03:43:02 +0000 UTC" deadline="2028-05-18 18:37:31.72899817 +0000 UTC" Sep 07 03:48:02.870281 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:02.870235 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14870h49m28.85876786s" Sep 07 03:48:02.881387 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.881362 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:02.981529 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:02.981496 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:03.000642 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.000607 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 03:48:03.001886 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.001858 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientMemory" Sep 07 03:48:03.001986 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.001893 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 03:48:03.001986 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.001903 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeHasSufficientPID" Sep 07 03:48:03.054453 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:03.054413 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed84f1ba21f69b974e5fd2991bf7d19.slice/crio-01a2858a66eabb3d96a9bdd9f2e5989cd57efa2b313f4b796327155337f4597a WatchSource:0}: Error finding container 01a2858a66eabb3d96a9bdd9f2e5989cd57efa2b313f4b796327155337f4597a: Status 404 returned error can't find the container with id 01a2858a66eabb3d96a9bdd9f2e5989cd57efa2b313f4b796327155337f4597a Sep 07 03:48:03.061922 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.061898 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 03:48:03.065204 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:03.065181 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62584f24accf83230cac126e1538660.slice/crio-ad4c299e7de3195d15deb026623d7a2b38e2db196592a3c40edf4ab43294f56f WatchSource:0}: Error finding container ad4c299e7de3195d15deb026623d7a2b38e2db196592a3c40edf4ab43294f56f: Status 404 returned error can't find the container with id ad4c299e7de3195d15deb026623d7a2b38e2db196592a3c40edf4ab43294f56f Sep 07 03:48:03.081736 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:03.081712 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:03.182777 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:03.182678 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:03.243133 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.243109 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 03:48:03.283405 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:03.283368 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-215.ec2.internal\" not found" Sep 07 03:48:03.381296 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.381251 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 03:48:03.381859 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.381838 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 03:48:03.440534 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.440437 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal" Sep 07 03:48:03.448232 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.448214 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 03:48:03.449347 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.449333 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" Sep 07 03:48:03.460674 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.460651 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 03:48:03.821401 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.821328 2555 apiserver.go:52] "Watching apiserver" Sep 07 03:48:03.827848 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.827780 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 07 03:48:03.830371 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.830343 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal","openshift-multus/multus-additional-cni-plugins-wcg5f","openshift-multus/multus-vg569","openshift-network-diagnostics/network-check-target-x6689","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs","openshift-cluster-node-tuning-operator/tuned-bgqnw","openshift-dns/node-resolver-6wvfr","openshift-multus/network-metrics-daemon-sxp6n","openshift-network-operator/iptables-alerter-9bpk4","openshift-ovn-kubernetes/ovnkube-node-xfkc8","kube-system/global-pull-secret-syncer-gbx7k","kube-system/konnectivity-agent-hp2z7","kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal","openshift-image-registry/node-ca-fst8d"] Sep 07 03:48:03.833180 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.833159 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:03.835993 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.835972 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ts8xw\"" Sep 07 03:48:03.836153 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.836122 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 07 03:48:03.836230 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.835974 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 07 03:48:03.836748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.836725 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:03.836856 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:03.836819 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:03.839697 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.839678 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 07 03:48:03.841942 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.841925 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vg569" Sep 07 03:48:03.845478 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.844552 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 07 03:48:03.845478 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.844568 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 07 03:48:03.845478 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.844950 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 07 03:48:03.845478 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.845037 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 07 03:48:03.845478 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.845183 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qzxbr\"" Sep 07 03:48:03.845478 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.845380 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:03.845478 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:03.845439 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:03.847656 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.847638 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849251 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-run-multus-certs\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849296 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8h4\" (UniqueName: \"kubernetes.io/projected/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-kube-api-access-lx8h4\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849323 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-system-cni-dir\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849348 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-var-lib-kubelet\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849378 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b83fecd8-da21-48bf-a8f4-13c33d46742a-hosts-file\") pod \"node-resolver-6wvfr\" (UID: \"b83fecd8-da21-48bf-a8f4-13c33d46742a\") " pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849402 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-cni-dir\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849426 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-run-k8s-cni-cncf-io\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849451 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-var-lib-cni-multus\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849476 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-hostroot\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849504 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-etc-kubernetes\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849530 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-socket-dir-parent\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849555 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b83fecd8-da21-48bf-a8f4-13c33d46742a-tmp-dir\") pod \"node-resolver-6wvfr\" (UID: \"b83fecd8-da21-48bf-a8f4-13c33d46742a\") " pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849580 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxftx\" (UniqueName: \"kubernetes.io/projected/b83fecd8-da21-48bf-a8f4-13c33d46742a-kube-api-access-fxftx\") pod \"node-resolver-6wvfr\" (UID: \"b83fecd8-da21-48bf-a8f4-13c33d46742a\") " pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849605 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a784083-f7eb-4508-9ef5-07ae8c51a973-kubelet-config\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849630 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a784083-f7eb-4508-9ef5-07ae8c51a973-dbus\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849676 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-cnibin\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.850878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849700 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-cni-binary-copy\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849727 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849754 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-run-netns\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849777 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-var-lib-cni-bin\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849832 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-conf-dir\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849856 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-daemon-config\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.849980 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.850119 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.850266 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j2hw7\"" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.850440 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.850459 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 07 03:48:03.851905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.850611 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.852975 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.852953 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 07 03:48:03.853276 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.853260 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:03.853683 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.853532 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 07 03:48:03.854308 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.854014 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8jhzj\"" Sep 07 03:48:03.855525 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.855487 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 07 03:48:03.855696 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.855645 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qqvgn\"" Sep 07 03:48:03.855779 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.855726 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:03.860045 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.860020 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 07 03:48:03.860900 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.860300 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:03.860900 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.860346 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 07 03:48:03.860900 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.860589 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 07 03:48:03.860900 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.860729 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dwtc8\"" Sep 07 03:48:03.861267 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.861249 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 07 03:48:03.861594 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.861572 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.862770 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.862743 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-25jrp\"" Sep 07 03:48:03.862956 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.862935 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 07 03:48:03.863169 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.863146 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 07 03:48:03.864081 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.864056 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:03.864214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.864088 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 07 03:48:03.864214 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:03.864162 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:03.864431 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.864411 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 07 03:48:03.864922 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.864673 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 07 03:48:03.864922 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.864712 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 07 03:48:03.865062 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.864926 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 07 03:48:03.865062 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.865012 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 07 03:48:03.865186 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.865114 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t2p9j\"" Sep 07 03:48:03.865598 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.865578 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 07 03:48:03.868169 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.867342 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:03.869839 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.869814 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 07 03:48:03.869945 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.869854 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 07 03:48:03.870047 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.870027 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9hhbp\"" Sep 07 03:48:03.875743 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.875717 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l6qn6" Sep 07 03:48:03.885262 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.884532 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l6qn6" Sep 07 03:48:03.935744 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.935644 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" event={"ID":"e62584f24accf83230cac126e1538660","Type":"ContainerStarted","Data":"ad4c299e7de3195d15deb026623d7a2b38e2db196592a3c40edf4ab43294f56f"} Sep 07 03:48:03.937722 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.937692 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal" event={"ID":"bed84f1ba21f69b974e5fd2991bf7d19","Type":"ContainerStarted","Data":"01a2858a66eabb3d96a9bdd9f2e5989cd57efa2b313f4b796327155337f4597a"} Sep 07 03:48:03.942017 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.941995 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 07 03:48:03.950542 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950517 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-ovnkube-script-lib\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.950644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950556 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-kubernetes\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.950644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950585 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkw9r\" (UniqueName: \"kubernetes.io/projected/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-kube-api-access-zkw9r\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:03.950644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950611 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-cni-bin\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.950816 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950660 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-systemd\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.950816 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950702 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b83fecd8-da21-48bf-a8f4-13c33d46742a-hosts-file\") pod \"node-resolver-6wvfr\" (UID: \"b83fecd8-da21-48bf-a8f4-13c33d46742a\") " pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:03.950816 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950741 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-run-k8s-cni-cncf-io\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.950816 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950771 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-var-lib-cni-multus\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.950816 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950796 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-cni-binary-copy\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:03.950816 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950809 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b83fecd8-da21-48bf-a8f4-13c33d46742a-hosts-file\") pod \"node-resolver-6wvfr\" (UID: \"b83fecd8-da21-48bf-a8f4-13c33d46742a\") " pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950821 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-lib-modules\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950843 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-run-k8s-cni-cncf-io\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950866 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-898p9\" (UniqueName: \"kubernetes.io/projected/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-kube-api-access-898p9\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950875 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-var-lib-cni-multus\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950901 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a784083-f7eb-4508-9ef5-07ae8c51a973-dbus\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950927 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-cnibin\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950952 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-run-netns\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.950991 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-daemon-config\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951017 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-cnibin\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951021 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-run-ovn\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951049 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zk4m\" (UniqueName: \"kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m\") pod \"network-check-target-x6689\" (UID: \"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6\") " pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951055 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-run-netns\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951090 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951114 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-cni-netd\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.951137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951140 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a784083-f7eb-4508-9ef5-07ae8c51a973-dbus\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951354 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f834b816-08bb-466b-8b19-d1af19636f85-konnectivity-ca\") pod \"konnectivity-agent-hp2z7\" (UID: \"f834b816-08bb-466b-8b19-d1af19636f85\") " pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:03.951358 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951373 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951388 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-socket-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951423 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-run\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:03.951465 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret podName:4a784083-f7eb-4508-9ef5-07ae8c51a973 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:04.451430234 +0000 UTC m=+3.026296511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret") pod "global-pull-secret-syncer-gbx7k" (UID: "4a784083-f7eb-4508-9ef5-07ae8c51a973") : object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951517 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lqr\" (UniqueName: \"kubernetes.io/projected/63e8d9bb-dce9-4c17-9854-47b9314f0217-kube-api-access-98lqr\") pod \"iptables-alerter-9bpk4\" (UID: \"63e8d9bb-dce9-4c17-9854-47b9314f0217\") " pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951553 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-var-lib-kubelet\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951578 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc82z\" (UniqueName: \"kubernetes.io/projected/ca804fd3-f751-4640-8e5a-a831ab9e2f55-kube-api-access-sc82z\") pod \"node-ca-fst8d\" (UID: \"ca804fd3-f751-4640-8e5a-a831ab9e2f55\") " pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951607 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-cni-dir\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951622 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-var-lib-kubelet\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951632 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-modprobe-d\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951658 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b83fecd8-da21-48bf-a8f4-13c33d46742a-tmp-dir\") pod \"node-resolver-6wvfr\" (UID: \"b83fecd8-da21-48bf-a8f4-13c33d46742a\") " pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951684 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-var-lib-cni-bin\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951709 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-cni-dir\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951737 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-conf-dir\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.951924 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951763 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca804fd3-f751-4640-8e5a-a831ab9e2f55-host\") pod \"node-ca-fst8d\" (UID: \"ca804fd3-f751-4640-8e5a-a831ab9e2f55\") " pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951769 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-var-lib-cni-bin\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951788 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-run-ovn-kubernetes\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951812 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-sys-fs\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951825 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-conf-dir\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951835 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63e8d9bb-dce9-4c17-9854-47b9314f0217-host-slash\") pod \"iptables-alerter-9bpk4\" (UID: \"63e8d9bb-dce9-4c17-9854-47b9314f0217\") " pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951870 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-system-cni-dir\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951907 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-etc-openvswitch\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951944 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-system-cni-dir\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951970 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca804fd3-f751-4640-8e5a-a831ab9e2f55-serviceca\") pod \"node-ca-fst8d\" (UID: \"ca804fd3-f751-4640-8e5a-a831ab9e2f55\") " pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.951997 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952024 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-node-log\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952050 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-ovn-node-metrics-cert\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952055 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-system-cni-dir\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952023 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b83fecd8-da21-48bf-a8f4-13c33d46742a-tmp-dir\") pod \"node-resolver-6wvfr\" (UID: \"b83fecd8-da21-48bf-a8f4-13c33d46742a\") " pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952089 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-device-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952168 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ksm6\" (UniqueName: \"kubernetes.io/projected/654e0e8a-3b0f-4263-8a5f-199214fff0bc-kube-api-access-5ksm6\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:03.952869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952194 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-sys\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952225 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-hostroot\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952251 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-run-netns\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952278 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-ovnkube-config\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952302 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2skg\" (UniqueName: \"kubernetes.io/projected/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-kube-api-access-h2skg\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952327 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63e8d9bb-dce9-4c17-9854-47b9314f0217-iptables-alerter-script\") pod \"iptables-alerter-9bpk4\" (UID: \"63e8d9bb-dce9-4c17-9854-47b9314f0217\") " pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952352 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-hostroot\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952353 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxftx\" (UniqueName: \"kubernetes.io/projected/b83fecd8-da21-48bf-a8f4-13c33d46742a-kube-api-access-fxftx\") pod \"node-resolver-6wvfr\" (UID: \"b83fecd8-da21-48bf-a8f4-13c33d46742a\") " pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952401 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a784083-f7eb-4508-9ef5-07ae8c51a973-kubelet-config\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952427 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-cni-binary-copy\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952453 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-run-multus-certs\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952481 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-var-lib-openvswitch\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952506 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-host\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952532 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952576 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-systemd-units\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952604 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952647 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a784083-f7eb-4508-9ef5-07ae8c51a973-kubelet-config\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:03.953748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952707 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-host-run-multus-certs\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952632 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952733 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f834b816-08bb-466b-8b19-d1af19636f85-agent-certs\") pod \"konnectivity-agent-hp2z7\" (UID: \"f834b816-08bb-466b-8b19-d1af19636f85\") " pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952751 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-registration-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952766 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952780 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-sysconfig\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952805 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-cnibin\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952821 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-log-socket\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952835 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-sysctl-d\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952848 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-var-lib-kubelet\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952867 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-etc-kubernetes\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952881 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952906 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-slash\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952925 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-run-systemd\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952939 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-env-overrides\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952954 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xl9r\" (UniqueName: \"kubernetes.io/projected/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-kube-api-access-5xl9r\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:03.954644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952968 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-tmp\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.955350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.952985 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-socket-dir-parent\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.955350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.953004 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8h4\" (UniqueName: \"kubernetes.io/projected/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-kube-api-access-lx8h4\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.955350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.953020 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-kubelet\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.955350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.953034 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-run-openvswitch\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:03.955350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.953049 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-sysctl-conf\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.955350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.953062 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-tuned\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:03.955350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.953198 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-cni-binary-copy\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.955350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.953222 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-daemon-config\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.955350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.953231 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-etc-kubernetes\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.955350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.953240 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-multus-socket-dir-parent\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:03.962462 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.962435 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 07 03:48:03.967810 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.967628 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxftx\" (UniqueName: \"kubernetes.io/projected/b83fecd8-da21-48bf-a8f4-13c33d46742a-kube-api-access-fxftx\") pod \"node-resolver-6wvfr\" (UID: \"b83fecd8-da21-48bf-a8f4-13c33d46742a\") " pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:03.968611 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:03.968579 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8h4\" (UniqueName: \"kubernetes.io/projected/cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9-kube-api-access-lx8h4\") pod \"multus-vg569\" (UID: \"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9\") " pod="openshift-multus/multus-vg569" Sep 07 03:48:04.020975 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.020942 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 03:48:04.053572 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053533 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-run-ovn\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.053572 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053570 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk4m\" (UniqueName: \"kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m\") pod \"network-check-target-x6689\" (UID: \"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6\") " pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:04.053818 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053606 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-cni-netd\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.053818 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053629 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f834b816-08bb-466b-8b19-d1af19636f85-konnectivity-ca\") pod \"konnectivity-agent-hp2z7\" (UID: \"f834b816-08bb-466b-8b19-d1af19636f85\") " pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:04.053818 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053644 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.053818 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053658 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-socket-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.053818 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053680 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-run\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.053818 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053797 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-run\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.054144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053843 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.054144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053901 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-run-ovn\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.054144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053943 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-socket-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.054144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.053966 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lqr\" (UniqueName: \"kubernetes.io/projected/63e8d9bb-dce9-4c17-9854-47b9314f0217-kube-api-access-98lqr\") pod \"iptables-alerter-9bpk4\" (UID: \"63e8d9bb-dce9-4c17-9854-47b9314f0217\") " pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:04.054144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054008 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc82z\" (UniqueName: \"kubernetes.io/projected/ca804fd3-f751-4640-8e5a-a831ab9e2f55-kube-api-access-sc82z\") pod \"node-ca-fst8d\" (UID: \"ca804fd3-f751-4640-8e5a-a831ab9e2f55\") " pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:04.054144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054029 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-modprobe-d\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.054144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054049 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca804fd3-f751-4640-8e5a-a831ab9e2f55-host\") pod \"node-ca-fst8d\" (UID: \"ca804fd3-f751-4640-8e5a-a831ab9e2f55\") " pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:04.054144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054072 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-run-ovn-kubernetes\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.054144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054107 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-sys-fs\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.054144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054131 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63e8d9bb-dce9-4c17-9854-47b9314f0217-host-slash\") pod \"iptables-alerter-9bpk4\" (UID: \"63e8d9bb-dce9-4c17-9854-47b9314f0217\") " pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054184 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-cni-netd\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054219 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63e8d9bb-dce9-4c17-9854-47b9314f0217-host-slash\") pod \"iptables-alerter-9bpk4\" (UID: \"63e8d9bb-dce9-4c17-9854-47b9314f0217\") " pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054255 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-system-cni-dir\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054187 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-system-cni-dir\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054281 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca804fd3-f751-4640-8e5a-a831ab9e2f55-host\") pod \"node-ca-fst8d\" (UID: \"ca804fd3-f751-4640-8e5a-a831ab9e2f55\") " pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054294 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-etc-openvswitch\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054322 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca804fd3-f751-4640-8e5a-a831ab9e2f55-serviceca\") pod \"node-ca-fst8d\" (UID: \"ca804fd3-f751-4640-8e5a-a831ab9e2f55\") " pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054326 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-run-ovn-kubernetes\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054354 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-etc-openvswitch\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054364 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-sys-fs\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054359 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054417 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-node-log\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054435 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-ovn-node-metrics-cert\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054452 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-device-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054473 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ksm6\" (UniqueName: \"kubernetes.io/projected/654e0e8a-3b0f-4263-8a5f-199214fff0bc-kube-api-access-5ksm6\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054377 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f834b816-08bb-466b-8b19-d1af19636f85-konnectivity-ca\") pod \"konnectivity-agent-hp2z7\" (UID: \"f834b816-08bb-466b-8b19-d1af19636f85\") " pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:04.054685 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054296 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-modprobe-d\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054497 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-sys\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054538 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-sys\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054541 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-run-netns\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054551 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-node-log\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054574 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-ovnkube-config\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054578 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-run-netns\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054612 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2skg\" (UniqueName: \"kubernetes.io/projected/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-kube-api-access-h2skg\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054640 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-device-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054642 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63e8d9bb-dce9-4c17-9854-47b9314f0217-iptables-alerter-script\") pod \"iptables-alerter-9bpk4\" (UID: \"63e8d9bb-dce9-4c17-9854-47b9314f0217\") " pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054694 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-var-lib-openvswitch\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054718 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-host\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054745 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054772 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-systemd-units\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054790 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca804fd3-f751-4640-8e5a-a831ab9e2f55-serviceca\") pod \"node-ca-fst8d\" (UID: \"ca804fd3-f751-4640-8e5a-a831ab9e2f55\") " pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054801 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054838 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:04.055593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054849 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054865 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f834b816-08bb-466b-8b19-d1af19636f85-agent-certs\") pod \"konnectivity-agent-hp2z7\" (UID: \"f834b816-08bb-466b-8b19-d1af19636f85\") " pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054892 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-registration-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054895 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054918 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054930 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-host\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054944 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-sysconfig\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054972 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-cnibin\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054996 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-log-socket\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055021 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-sysctl-d\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055046 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-var-lib-kubelet\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055073 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055102 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-slash\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055147 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-run-systemd\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055173 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-env-overrides\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054971 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-systemd-units\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055198 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xl9r\" (UniqueName: \"kubernetes.io/projected/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-kube-api-access-5xl9r\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:04.056498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.054892 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-var-lib-openvswitch\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055243 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-ovnkube-config\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055273 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-log-socket\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055281 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63e8d9bb-dce9-4c17-9854-47b9314f0217-iptables-alerter-script\") pod \"iptables-alerter-9bpk4\" (UID: \"63e8d9bb-dce9-4c17-9854-47b9314f0217\") " pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055314 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.055322 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055332 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-sysconfig\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055347 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-var-lib-kubelet\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055370 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-cnibin\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.055383 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs podName:9d176a6b-c22a-4f6a-9551-1dbd25928cf8 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:04.555364069 +0000 UTC m=+3.130230356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs") pod "network-metrics-daemon-sxp6n" (UID: "9d176a6b-c22a-4f6a-9551-1dbd25928cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055400 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055409 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/654e0e8a-3b0f-4263-8a5f-199214fff0bc-registration-dir\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055424 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-run-systemd\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055426 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-tmp\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055433 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055441 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-sysctl-d\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055465 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-kubelet\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.057384 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055484 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-slash\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055494 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-run-openvswitch\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055536 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-run-openvswitch\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055589 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-sysctl-conf\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055591 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-kubelet\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055616 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-tuned\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055639 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-ovnkube-script-lib\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055702 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-kubernetes\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055722 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-sysctl-conf\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055776 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-kubernetes\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.055891 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-env-overrides\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056039 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkw9r\" (UniqueName: \"kubernetes.io/projected/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-kube-api-access-zkw9r\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056077 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-cni-bin\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056116 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-systemd\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056145 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-cni-binary-copy\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056168 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-ovnkube-script-lib\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056169 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-lib-modules\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.058278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056179 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-host-cni-bin\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.059021 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056219 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-898p9\" (UniqueName: \"kubernetes.io/projected/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-kube-api-access-898p9\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.059021 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056229 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-systemd\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.059021 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056322 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-lib-modules\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.059021 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.056689 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-cni-binary-copy\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.059021 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.057798 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-ovn-node-metrics-cert\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.059021 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.058215 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-tmp\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.059021 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.058401 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f834b816-08bb-466b-8b19-d1af19636f85-agent-certs\") pod \"konnectivity-agent-hp2z7\" (UID: \"f834b816-08bb-466b-8b19-d1af19636f85\") " pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:04.059021 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.058601 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-etc-tuned\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.060036 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.060012 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 03:48:04.060036 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.060039 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 03:48:04.060237 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.060054 2555 projected.go:196] Error preparing data for projected volume kube-api-access-7zk4m for pod openshift-network-diagnostics/network-check-target-x6689: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:04.060237 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.060137 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m podName:1c70ff1e-fc11-4f58-80b5-98a9901fd7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:04.560117913 +0000 UTC m=+3.134984187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7zk4m" (UniqueName: "kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m") pod "network-check-target-x6689" (UID: "1c70ff1e-fc11-4f58-80b5-98a9901fd7d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:04.062561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.062537 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lqr\" (UniqueName: \"kubernetes.io/projected/63e8d9bb-dce9-4c17-9854-47b9314f0217-kube-api-access-98lqr\") pod \"iptables-alerter-9bpk4\" (UID: \"63e8d9bb-dce9-4c17-9854-47b9314f0217\") " pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:04.063662 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.063618 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ksm6\" (UniqueName: \"kubernetes.io/projected/654e0e8a-3b0f-4263-8a5f-199214fff0bc-kube-api-access-5ksm6\") pod \"aws-ebs-csi-driver-node-djbrs\" (UID: \"654e0e8a-3b0f-4263-8a5f-199214fff0bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.064366 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.064342 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2skg\" (UniqueName: \"kubernetes.io/projected/e6fbaee3-10f8-46cf-8e6e-656c95e7592b-kube-api-access-h2skg\") pod \"tuned-bgqnw\" (UID: \"e6fbaee3-10f8-46cf-8e6e-656c95e7592b\") " pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.064907 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.064850 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xl9r\" (UniqueName: \"kubernetes.io/projected/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-kube-api-access-5xl9r\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:04.065359 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.065329 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-898p9\" (UniqueName: \"kubernetes.io/projected/b2633695-8fd3-426b-b6e6-d2a150ffbd7c-kube-api-access-898p9\") pod \"ovnkube-node-xfkc8\" (UID: \"b2633695-8fd3-426b-b6e6-d2a150ffbd7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.066068 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.066035 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkw9r\" (UniqueName: \"kubernetes.io/projected/1231ac90-f33a-4adb-bfb4-8bd0b14c9348-kube-api-access-zkw9r\") pod \"multus-additional-cni-plugins-wcg5f\" (UID: \"1231ac90-f33a-4adb-bfb4-8bd0b14c9348\") " pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.066454 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.066431 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc82z\" (UniqueName: \"kubernetes.io/projected/ca804fd3-f751-4640-8e5a-a831ab9e2f55-kube-api-access-sc82z\") pod \"node-ca-fst8d\" (UID: \"ca804fd3-f751-4640-8e5a-a831ab9e2f55\") " pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:04.147048 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.146962 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6wvfr" Sep 07 03:48:04.161999 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.161976 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vg569" Sep 07 03:48:04.177843 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.177813 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" Sep 07 03:48:04.180482 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.180459 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" Sep 07 03:48:04.188262 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.188244 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9bpk4" Sep 07 03:48:04.203018 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.202998 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fst8d" Sep 07 03:48:04.212953 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.212931 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" Sep 07 03:48:04.221033 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.221011 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:04.229052 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.229033 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:04.459277 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.459174 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:04.459448 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.459380 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:04.459448 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.459446 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret podName:4a784083-f7eb-4508-9ef5-07ae8c51a973 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:05.459431046 +0000 UTC m=+4.034297319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret") pod "global-pull-secret-syncer-gbx7k" (UID: "4a784083-f7eb-4508-9ef5-07ae8c51a973") : object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:04.560331 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.560292 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk4m\" (UniqueName: \"kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m\") pod \"network-check-target-x6689\" (UID: \"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6\") " pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:04.560539 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.560363 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:04.560539 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.560470 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:04.560539 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.560489 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 03:48:04.560539 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.560518 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 03:48:04.560539 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.560529 2555 projected.go:196] Error preparing data for projected volume kube-api-access-7zk4m for pod openshift-network-diagnostics/network-check-target-x6689: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:04.560539 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.560542 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs podName:9d176a6b-c22a-4f6a-9551-1dbd25928cf8 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:05.5605218 +0000 UTC m=+4.135388073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs") pod "network-metrics-daemon-sxp6n" (UID: "9d176a6b-c22a-4f6a-9551-1dbd25928cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:04.560865 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.560582 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m podName:1c70ff1e-fc11-4f58-80b5-98a9901fd7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:05.560569677 +0000 UTC m=+4.135435945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zk4m" (UniqueName: "kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m") pod "network-check-target-x6689" (UID: "1c70ff1e-fc11-4f58-80b5-98a9901fd7d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:04.889807 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.889725 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 03:43:04 +0000 UTC" deadline="2028-06-15 11:15:30.183819292 +0000 UTC" Sep 07 03:48:04.889807 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.889772 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15535h27m25.294056731s" Sep 07 03:48:04.931145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:04.931111 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:04.931322 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:04.931255 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:05.237587 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:05.237549 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd01acdb_295d_4cb0_ac8c_e0db0f5de8e9.slice/crio-8980c6726c025609d49d189e1eaea715fd9c79747727fa8f1f7f995f66c38648 WatchSource:0}: Error finding container 8980c6726c025609d49d189e1eaea715fd9c79747727fa8f1f7f995f66c38648: Status 404 returned error can't find the container with id 8980c6726c025609d49d189e1eaea715fd9c79747727fa8f1f7f995f66c38648 Sep 07 03:48:05.247482 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:05.247350 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1231ac90_f33a_4adb_bfb4_8bd0b14c9348.slice/crio-66f4485708c754c3edfef5f86882eeb2aa45febfb8b011473e363fa8a2a4ac6d WatchSource:0}: Error finding container 66f4485708c754c3edfef5f86882eeb2aa45febfb8b011473e363fa8a2a4ac6d: Status 404 returned error can't find the container with id 66f4485708c754c3edfef5f86882eeb2aa45febfb8b011473e363fa8a2a4ac6d Sep 07 03:48:05.270438 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:05.270404 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca804fd3_f751_4640_8e5a_a831ab9e2f55.slice/crio-0e72072804fced1240b91d7906c31a0cbe6604cb69509af1a86f8e59f40db935 WatchSource:0}: Error finding container 0e72072804fced1240b91d7906c31a0cbe6604cb69509af1a86f8e59f40db935: Status 404 returned error can't find the container with id 0e72072804fced1240b91d7906c31a0cbe6604cb69509af1a86f8e59f40db935 Sep 07 03:48:05.271671 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:05.271644 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf834b816_08bb_466b_8b19_d1af19636f85.slice/crio-7bad6e05e8c275b445aa209c8c868a7ce52d784e4f176d7d026bc52d68337197 WatchSource:0}: Error finding container 7bad6e05e8c275b445aa209c8c868a7ce52d784e4f176d7d026bc52d68337197: Status 404 returned error can't find the container with id 7bad6e05e8c275b445aa209c8c868a7ce52d784e4f176d7d026bc52d68337197 Sep 07 03:48:05.272435 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:05.272410 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2633695_8fd3_426b_b6e6_d2a150ffbd7c.slice/crio-f7775717984f89287cf393890d19f9f29191beaf05792cb4d16b81461d0333aa WatchSource:0}: Error finding container f7775717984f89287cf393890d19f9f29191beaf05792cb4d16b81461d0333aa: Status 404 returned error can't find the container with id f7775717984f89287cf393890d19f9f29191beaf05792cb4d16b81461d0333aa Sep 07 03:48:05.466698 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.466662 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:05.466857 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:05.466823 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:05.466900 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:05.466892 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret podName:4a784083-f7eb-4508-9ef5-07ae8c51a973 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:07.466875399 +0000 UTC m=+6.041741673 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret") pod "global-pull-secret-syncer-gbx7k" (UID: "4a784083-f7eb-4508-9ef5-07ae8c51a973") : object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:05.567658 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.567603 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:05.567813 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.567665 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk4m\" (UniqueName: \"kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m\") pod \"network-check-target-x6689\" (UID: \"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6\") " pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:05.567813 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:05.567746 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:05.567813 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:05.567788 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 03:48:05.567813 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:05.567803 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 03:48:05.567813 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:05.567813 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs podName:9d176a6b-c22a-4f6a-9551-1dbd25928cf8 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:07.567792038 +0000 UTC m=+6.142658327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs") pod "network-metrics-daemon-sxp6n" (UID: "9d176a6b-c22a-4f6a-9551-1dbd25928cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:05.567983 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:05.567815 2555 projected.go:196] Error preparing data for projected volume kube-api-access-7zk4m for pod openshift-network-diagnostics/network-check-target-x6689: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:05.567983 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:05.567854 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m podName:1c70ff1e-fc11-4f58-80b5-98a9901fd7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:07.567842883 +0000 UTC m=+6.142709167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zk4m" (UniqueName: "kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m") pod "network-check-target-x6689" (UID: "1c70ff1e-fc11-4f58-80b5-98a9901fd7d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:05.891003 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.890883 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 03:43:04 +0000 UTC" deadline="2028-05-02 21:15:55.929705997 +0000 UTC" Sep 07 03:48:05.891003 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.890956 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14489h27m50.038755001s" Sep 07 03:48:05.934891 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.934263 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:05.934891 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:05.934406 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:05.935507 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.935298 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:05.935507 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:05.935406 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:05.947486 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.947364 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" event={"ID":"b2633695-8fd3-426b-b6e6-d2a150ffbd7c","Type":"ContainerStarted","Data":"f7775717984f89287cf393890d19f9f29191beaf05792cb4d16b81461d0333aa"} Sep 07 03:48:05.955324 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.955219 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fst8d" event={"ID":"ca804fd3-f751-4640-8e5a-a831ab9e2f55","Type":"ContainerStarted","Data":"0e72072804fced1240b91d7906c31a0cbe6604cb69509af1a86f8e59f40db935"} Sep 07 03:48:05.979180 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.979124 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" event={"ID":"654e0e8a-3b0f-4263-8a5f-199214fff0bc","Type":"ContainerStarted","Data":"7bf2f017b10cc7a7d4a31f51a544c5ace7595aed38dc40bbcd7d074dab37586f"} Sep 07 03:48:05.985205 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.985056 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9bpk4" event={"ID":"63e8d9bb-dce9-4c17-9854-47b9314f0217","Type":"ContainerStarted","Data":"44dcbac6e69f4a820b8feb586eecb39c3b241efbfb6835c5ab4f69230598eca8"} Sep 07 03:48:05.989600 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.989523 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6wvfr" event={"ID":"b83fecd8-da21-48bf-a8f4-13c33d46742a","Type":"ContainerStarted","Data":"08beaa576112dd7a1002fff8eab81039d9f3b30ddbc959513a53fa7454058f02"} Sep 07 03:48:05.998565 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:05.998536 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" event={"ID":"e6fbaee3-10f8-46cf-8e6e-656c95e7592b","Type":"ContainerStarted","Data":"9c3c62cd4951330dafd6bc1408a2e8fefc25ae3f54d6aca593234c60ce2bebbd"} Sep 07 03:48:06.007009 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:06.006972 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vg569" event={"ID":"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9","Type":"ContainerStarted","Data":"8980c6726c025609d49d189e1eaea715fd9c79747727fa8f1f7f995f66c38648"} Sep 07 03:48:06.025944 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:06.025905 2555 generic.go:356] "Generic (PLEG): container finished" podID="e62584f24accf83230cac126e1538660" containerID="c6ecece4982874bcb34f9d6ed51427df0d4717c133a8c96d726fac8f44c52e32" exitCode=0 Sep 07 03:48:06.026126 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:06.026013 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" event={"ID":"e62584f24accf83230cac126e1538660","Type":"ContainerDied","Data":"c6ecece4982874bcb34f9d6ed51427df0d4717c133a8c96d726fac8f44c52e32"} Sep 07 03:48:06.034757 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:06.034723 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal" event={"ID":"bed84f1ba21f69b974e5fd2991bf7d19","Type":"ContainerStarted","Data":"46d17fb0f1582772b8b0784f935e82ed6d27fc773d0c5bf4e022973050fe0a74"} Sep 07 03:48:06.050749 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:06.050660 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hp2z7" event={"ID":"f834b816-08bb-466b-8b19-d1af19636f85","Type":"ContainerStarted","Data":"7bad6e05e8c275b445aa209c8c868a7ce52d784e4f176d7d026bc52d68337197"} Sep 07 03:48:06.056040 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:06.055969 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" event={"ID":"1231ac90-f33a-4adb-bfb4-8bd0b14c9348","Type":"ContainerStarted","Data":"66f4485708c754c3edfef5f86882eeb2aa45febfb8b011473e363fa8a2a4ac6d"} Sep 07 03:48:06.059428 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:06.058557 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-215.ec2.internal" podStartSLOduration=3.058536957 podStartE2EDuration="3.058536957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:06.058086469 +0000 UTC m=+4.632952763" watchObservedRunningTime="2026-09-07 03:48:06.058536957 +0000 UTC m=+4.633403252" Sep 07 03:48:06.931160 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:06.930609 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:06.931160 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:06.930756 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:07.080490 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:07.080192 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" event={"ID":"e62584f24accf83230cac126e1538660","Type":"ContainerStarted","Data":"6699af0c07d539df7d0f1291f467c1ed48a62756ebeabb2e4b8e3e38876a8242"} Sep 07 03:48:07.484612 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:07.484527 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:07.484781 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:07.484706 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:07.484781 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:07.484766 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret podName:4a784083-f7eb-4508-9ef5-07ae8c51a973 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:11.484748317 +0000 UTC m=+10.059614600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret") pod "global-pull-secret-syncer-gbx7k" (UID: "4a784083-f7eb-4508-9ef5-07ae8c51a973") : object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:07.594074 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:07.585774 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk4m\" (UniqueName: \"kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m\") pod \"network-check-target-x6689\" (UID: \"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6\") " pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:07.594074 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:07.585923 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:07.595023 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:07.594480 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 03:48:07.595023 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:07.594509 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 03:48:07.595023 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:07.594530 2555 projected.go:196] Error preparing data for projected volume kube-api-access-7zk4m for pod openshift-network-diagnostics/network-check-target-x6689: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:07.595023 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:07.594615 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m podName:1c70ff1e-fc11-4f58-80b5-98a9901fd7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:11.594576143 +0000 UTC m=+10.169442429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zk4m" (UniqueName: "kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m") pod "network-check-target-x6689" (UID: "1c70ff1e-fc11-4f58-80b5-98a9901fd7d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:07.595565 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:07.595252 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:07.595565 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:07.595319 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs podName:9d176a6b-c22a-4f6a-9551-1dbd25928cf8 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:11.595303814 +0000 UTC m=+10.170170082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs") pod "network-metrics-daemon-sxp6n" (UID: "9d176a6b-c22a-4f6a-9551-1dbd25928cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:07.931263 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:07.931184 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:07.931814 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:07.931307 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:07.931814 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:07.931807 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:07.931976 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:07.931953 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:08.930860 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:08.930819 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:08.931043 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:08.930948 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:09.930248 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:09.930216 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:09.930686 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:09.930352 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:09.930865 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:09.930844 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:09.930969 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:09.930949 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:10.931603 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:10.931119 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:10.931603 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:10.931269 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:11.521820 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:11.521781 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:11.521966 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:11.521926 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:11.522022 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:11.521990 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret podName:4a784083-f7eb-4508-9ef5-07ae8c51a973 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:19.52197188 +0000 UTC m=+18.096838172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret") pod "global-pull-secret-syncer-gbx7k" (UID: "4a784083-f7eb-4508-9ef5-07ae8c51a973") : object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:11.623125 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:11.623090 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:11.623294 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:11.623159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk4m\" (UniqueName: \"kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m\") pod \"network-check-target-x6689\" (UID: \"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6\") " pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:11.623294 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:11.623239 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:11.623294 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:11.623291 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 03:48:11.623443 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:11.623305 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 03:48:11.623443 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:11.623316 2555 projected.go:196] Error preparing data for projected volume kube-api-access-7zk4m for pod openshift-network-diagnostics/network-check-target-x6689: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:11.623443 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:11.623316 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs podName:9d176a6b-c22a-4f6a-9551-1dbd25928cf8 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:19.623296241 +0000 UTC m=+18.198162525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs") pod "network-metrics-daemon-sxp6n" (UID: "9d176a6b-c22a-4f6a-9551-1dbd25928cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:11.623443 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:11.623366 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m podName:1c70ff1e-fc11-4f58-80b5-98a9901fd7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:19.623350774 +0000 UTC m=+18.198217045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zk4m" (UniqueName: "kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m") pod "network-check-target-x6689" (UID: "1c70ff1e-fc11-4f58-80b5-98a9901fd7d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:11.931674 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:11.931103 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:11.931674 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:11.931114 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:11.931674 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:11.931246 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:11.931674 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:11.931282 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:12.930979 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:12.930944 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:12.931180 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:12.931109 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:13.930778 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:13.930743 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:13.931299 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:13.930902 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:13.931299 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:13.930996 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:13.931299 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:13.931088 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:14.930017 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:14.929988 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:14.930195 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:14.930118 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:15.930247 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:15.930212 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:15.930607 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:15.930212 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:15.930607 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:15.930321 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:15.930607 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:15.930391 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:16.931069 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:16.931021 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:16.931492 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:16.931152 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:17.930504 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:17.930462 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:17.930668 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:17.930525 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:17.930668 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:17.930607 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:17.930795 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:17.930728 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:18.930750 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:18.930721 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:18.931203 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:18.930856 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:19.584881 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:19.584843 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:19.585050 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:19.585029 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:19.585125 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:19.585117 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret podName:4a784083-f7eb-4508-9ef5-07ae8c51a973 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:35.585097973 +0000 UTC m=+34.159964252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret") pod "global-pull-secret-syncer-gbx7k" (UID: "4a784083-f7eb-4508-9ef5-07ae8c51a973") : object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:19.685320 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:19.685279 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk4m\" (UniqueName: \"kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m\") pod \"network-check-target-x6689\" (UID: \"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6\") " pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:19.685495 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:19.685344 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:19.685495 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:19.685462 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:19.685613 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:19.685511 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 03:48:19.685613 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:19.685538 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 03:48:19.685613 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:19.685550 2555 projected.go:196] Error preparing data for projected volume kube-api-access-7zk4m for pod openshift-network-diagnostics/network-check-target-x6689: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:19.685613 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:19.685523 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs podName:9d176a6b-c22a-4f6a-9551-1dbd25928cf8 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:35.685508959 +0000 UTC m=+34.260375225 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs") pod "network-metrics-daemon-sxp6n" (UID: "9d176a6b-c22a-4f6a-9551-1dbd25928cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:19.685613 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:19.685604 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m podName:1c70ff1e-fc11-4f58-80b5-98a9901fd7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:35.685588303 +0000 UTC m=+34.260454584 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zk4m" (UniqueName: "kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m") pod "network-check-target-x6689" (UID: "1c70ff1e-fc11-4f58-80b5-98a9901fd7d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:19.930497 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:19.930413 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:19.930662 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:19.930550 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:19.930662 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:19.930584 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:19.930662 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:19.930638 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:20.930259 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:20.930221 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:20.930722 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:20.930360 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:21.931920 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:21.931891 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:21.932327 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:21.931985 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:21.932327 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:21.932026 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:21.932327 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:21.932072 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:22.930362 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:22.930331 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:22.930526 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:22.930430 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:23.930806 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:23.930629 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:23.931525 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:23.930687 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:23.931525 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:23.930888 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:23.931525 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:23.930961 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:24.124883 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.124808 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" event={"ID":"b2633695-8fd3-426b-b6e6-d2a150ffbd7c","Type":"ContainerStarted","Data":"e0b6a234f5d08a9b088e55d42284822ef2d4c3c7062c74b2b00c6047d4a526ce"} Sep 07 03:48:24.125104 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.124886 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" event={"ID":"b2633695-8fd3-426b-b6e6-d2a150ffbd7c","Type":"ContainerStarted","Data":"1ff48667a408d3f07ee99eda60138c132ce2134b09b61d97dc121dba0ef9c77d"} Sep 07 03:48:24.125104 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.124934 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" event={"ID":"b2633695-8fd3-426b-b6e6-d2a150ffbd7c","Type":"ContainerStarted","Data":"8c6ec52efe3817812f62b50a46a7416f9089cc74060064b7ea65e1c4252fbc6e"} Sep 07 03:48:24.125104 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.124977 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" event={"ID":"b2633695-8fd3-426b-b6e6-d2a150ffbd7c","Type":"ContainerStarted","Data":"e5aac1dba97f48565624303c9ebadb9d85ba3914c5889eaa9d54bdfa13980ba1"} Sep 07 03:48:24.125104 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.125019 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" event={"ID":"b2633695-8fd3-426b-b6e6-d2a150ffbd7c","Type":"ContainerStarted","Data":"f6146434817eeeff05570d0e4f0bdc5f5eee2433828e102af635b44cb27c624d"} Sep 07 03:48:24.125104 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.125081 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" event={"ID":"b2633695-8fd3-426b-b6e6-d2a150ffbd7c","Type":"ContainerStarted","Data":"1de157f22a74f999ca4fc73fc3a180c16a9fa86c659f7c2b549325b389b4101a"} Sep 07 03:48:24.126087 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.126050 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fst8d" event={"ID":"ca804fd3-f751-4640-8e5a-a831ab9e2f55","Type":"ContainerStarted","Data":"44f9229f8d1085596537f588b45d076f629c38319cc80cc15dc5e700b99ad77b"} Sep 07 03:48:24.127387 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.127362 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" event={"ID":"654e0e8a-3b0f-4263-8a5f-199214fff0bc","Type":"ContainerStarted","Data":"03ac4fbc58dfeb89006c31d38ca30b222326fc69dd6407c450a6e9c9db4569be"} Sep 07 03:48:24.128638 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.128617 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6wvfr" event={"ID":"b83fecd8-da21-48bf-a8f4-13c33d46742a","Type":"ContainerStarted","Data":"cc7fa912b1a3c7f4fc67b2e2e10d39c380238e04119a229c37ca70a6be6403a2"} Sep 07 03:48:24.130279 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.130262 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" event={"ID":"e6fbaee3-10f8-46cf-8e6e-656c95e7592b","Type":"ContainerStarted","Data":"09d3f3f952ee527b543e4fb6be69c43c05b516f0b5e26c3332f2942731cbaf6d"} Sep 07 03:48:24.131684 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.131664 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vg569" event={"ID":"cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9","Type":"ContainerStarted","Data":"d4248765eb25651520dadafbf9150a213f51bca6d33e8a00535602a0511c05bd"} Sep 07 03:48:24.132994 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.132976 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hp2z7" event={"ID":"f834b816-08bb-466b-8b19-d1af19636f85","Type":"ContainerStarted","Data":"7cad8d2ee61b36c8c8a839491b0324185059999f2063ac0a6a75d4638fddb3c0"} Sep 07 03:48:24.134475 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.134446 2555 generic.go:356] "Generic (PLEG): container finished" podID="1231ac90-f33a-4adb-bfb4-8bd0b14c9348" containerID="9b8e5be193219f94d835819e8b7bd5b15fe835288ccb657fd4a63371b704873b" exitCode=0 Sep 07 03:48:24.134556 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.134483 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" event={"ID":"1231ac90-f33a-4adb-bfb4-8bd0b14c9348","Type":"ContainerDied","Data":"9b8e5be193219f94d835819e8b7bd5b15fe835288ccb657fd4a63371b704873b"} Sep 07 03:48:24.140147 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.140113 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-215.ec2.internal" podStartSLOduration=21.140102105 podStartE2EDuration="21.140102105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:07.097293655 +0000 UTC m=+5.672159966" watchObservedRunningTime="2026-09-07 03:48:24.140102105 +0000 UTC m=+22.714968394" Sep 07 03:48:24.153253 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.153219 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hp2z7" podStartSLOduration=4.486221852 podStartE2EDuration="22.15320742s" totalImagesPullingTime="17.666985568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:24.152811952 +0000 UTC m=+22.727678233" watchObservedRunningTime="2026-09-07 03:48:24.15320742 +0000 UTC m=+22.728073709" Sep 07 03:48:24.153371 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.153337 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fst8d" podStartSLOduration=4.442716038 podStartE2EDuration="22.15332928s" totalImagesPullingTime="17.710613242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:24.140102119 +0000 UTC m=+22.714968409" watchObservedRunningTime="2026-09-07 03:48:24.15332928 +0000 UTC m=+22.728195570" Sep 07 03:48:24.169424 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.169385 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6wvfr" podStartSLOduration=4.419528729 podStartE2EDuration="22.169376116s" totalImagesPullingTime="17.749847387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:24.169312856 +0000 UTC m=+22.744179144" watchObservedRunningTime="2026-09-07 03:48:24.169376116 +0000 UTC m=+22.744242404" Sep 07 03:48:24.185978 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.185940 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vg569" podStartSLOduration=4.381128773 podStartE2EDuration="22.185929591s" totalImagesPullingTime="17.804800818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:24.185686823 +0000 UTC m=+22.760553111" watchObservedRunningTime="2026-09-07 03:48:24.185929591 +0000 UTC m=+22.760795879" Sep 07 03:48:24.224713 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.224681 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bgqnw" podStartSLOduration=4.4692768990000005 podStartE2EDuration="22.224671554s" totalImagesPullingTime="17.755394655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:24.22461815 +0000 UTC m=+22.799484449" watchObservedRunningTime="2026-09-07 03:48:24.224671554 +0000 UTC m=+22.799537842" Sep 07 03:48:24.528842 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.528822 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 07 03:48:24.899347 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.899206 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T03:48:24.528836353Z","UUID":"7d59def6-b643-4b36-a48e-28dc78034f81","Handler":null,"Name":"","Endpoint":""} Sep 07 03:48:24.900918 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.900898 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 07 03:48:24.901044 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.900926 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 07 03:48:24.930251 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:24.930228 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:24.930390 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:24.930368 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:25.139557 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:25.139522 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" event={"ID":"654e0e8a-3b0f-4263-8a5f-199214fff0bc","Type":"ContainerStarted","Data":"a9e4c954263c5c3e59315e39eda42d945ef1067ac6f5a4bda1bca9224ee3dca0"} Sep 07 03:48:25.141864 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:25.141787 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9bpk4" event={"ID":"63e8d9bb-dce9-4c17-9854-47b9314f0217","Type":"ContainerStarted","Data":"64fbfed163b2c5082140e5f26b90b5fa33fb51aadd2bc7d5ebbb8cb2be0a4083"} Sep 07 03:48:25.156376 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:25.156289 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9bpk4" podStartSLOduration=5.4391153 podStartE2EDuration="23.156270994s" totalImagesPullingTime="17.717155694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:25.155861049 +0000 UTC m=+23.730727359" watchObservedRunningTime="2026-09-07 03:48:25.156270994 +0000 UTC m=+23.731137284" Sep 07 03:48:25.930570 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:25.930542 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:25.930705 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:25.930671 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:25.930938 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:25.930903 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:25.931039 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:25.931000 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:26.146892 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:26.146816 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" event={"ID":"b2633695-8fd3-426b-b6e6-d2a150ffbd7c","Type":"ContainerStarted","Data":"42ff9ffed1be9eec8c70784d239ff660c2674969d3436ae2f481b12637498f36"} Sep 07 03:48:26.148802 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:26.148777 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" event={"ID":"654e0e8a-3b0f-4263-8a5f-199214fff0bc","Type":"ContainerStarted","Data":"a251422b93e0b51f787318aa99a7174ebd849c401567f7a10d5bd8a9caa3de20"} Sep 07 03:48:26.169097 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:26.169035 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djbrs" podStartSLOduration=3.987047254 podStartE2EDuration="24.169022472s" totalImagesPullingTime="20.181975218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:26.168674244 +0000 UTC m=+24.743540548" watchObservedRunningTime="2026-09-07 03:48:26.169022472 +0000 UTC m=+24.743888766" Sep 07 03:48:26.930352 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:26.930325 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:26.930519 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:26.930423 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:27.230036 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:27.229968 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:27.230616 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:27.230600 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:27.930322 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:27.930286 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:27.930496 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:27.930416 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:27.930562 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:27.930496 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:27.930653 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:27.930624 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:28.152124 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:28.152097 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:28.152627 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:28.152611 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hp2z7" Sep 07 03:48:28.930970 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:28.930801 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:28.931685 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:28.931098 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:29.156575 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:29.156544 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" event={"ID":"1231ac90-f33a-4adb-bfb4-8bd0b14c9348","Type":"ContainerStarted","Data":"591984591db4c624fd74d5ee7ff00ed6319573e7b1b12179d94f4a320b27956e"} Sep 07 03:48:29.160281 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:29.160256 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" event={"ID":"b2633695-8fd3-426b-b6e6-d2a150ffbd7c","Type":"ContainerStarted","Data":"f60aba34ea0c1707b01ee1333888517536fc206e7c3163d0ae7e53b72ad582ef"} Sep 07 03:48:29.160616 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:29.160587 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:29.160716 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:29.160644 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:29.177267 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:29.177245 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:29.177380 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:29.177371 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:29.205434 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:29.205395 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" podStartSLOduration=9.439661935 podStartE2EDuration="27.205380005s" totalImagesPullingTime="17.76571807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:29.205133728 +0000 UTC m=+27.780000007" watchObservedRunningTime="2026-09-07 03:48:29.205380005 +0000 UTC m=+27.780246294" Sep 07 03:48:29.933316 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:29.933281 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:29.933778 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:29.933296 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:29.933778 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:29.933378 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:29.933778 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:29.933454 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:30.164408 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:30.164371 2555 generic.go:356] "Generic (PLEG): container finished" podID="1231ac90-f33a-4adb-bfb4-8bd0b14c9348" containerID="591984591db4c624fd74d5ee7ff00ed6319573e7b1b12179d94f4a320b27956e" exitCode=0 Sep 07 03:48:30.164556 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:30.164446 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" event={"ID":"1231ac90-f33a-4adb-bfb4-8bd0b14c9348","Type":"ContainerDied","Data":"591984591db4c624fd74d5ee7ff00ed6319573e7b1b12179d94f4a320b27956e"} Sep 07 03:48:30.164628 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:30.164570 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 07 03:48:30.558907 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:30.558870 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:48:30.632326 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:30.632250 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x6689"] Sep 07 03:48:30.632469 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:30.632356 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:30.632469 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:30.632433 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:30.635960 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:30.635933 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sxp6n"] Sep 07 03:48:30.636120 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:30.636106 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:30.636235 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:30.636217 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:30.636498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:30.636477 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gbx7k"] Sep 07 03:48:30.636562 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:30.636548 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:30.636645 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:30.636626 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:31.931240 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:31.931211 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:31.931585 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:31.931326 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:31.931585 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:31.931318 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:31.931585 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:31.931431 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:32.170947 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:32.170906 2555 generic.go:356] "Generic (PLEG): container finished" podID="1231ac90-f33a-4adb-bfb4-8bd0b14c9348" containerID="33bf8a3587945a6536c2a7e4fa1362fa1dc9ec4f03d0b3845c1e6dff6ad38d52" exitCode=0 Sep 07 03:48:32.171090 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:32.170991 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" event={"ID":"1231ac90-f33a-4adb-bfb4-8bd0b14c9348","Type":"ContainerDied","Data":"33bf8a3587945a6536c2a7e4fa1362fa1dc9ec4f03d0b3845c1e6dff6ad38d52"} Sep 07 03:48:32.930960 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:32.930922 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:32.931112 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:32.931030 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:33.175694 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:33.175664 2555 generic.go:356] "Generic (PLEG): container finished" podID="1231ac90-f33a-4adb-bfb4-8bd0b14c9348" containerID="39a378be812f6dadf44c35c974806ad9a8cf2afb38261e1201aef390bafde527" exitCode=0 Sep 07 03:48:33.176027 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:33.175699 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" event={"ID":"1231ac90-f33a-4adb-bfb4-8bd0b14c9348","Type":"ContainerDied","Data":"39a378be812f6dadf44c35c974806ad9a8cf2afb38261e1201aef390bafde527"} Sep 07 03:48:33.930479 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:33.930445 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:33.930628 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:33.930602 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:33.930682 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:33.930647 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:33.930738 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:33.930722 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:34.930868 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:34.930798 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:34.931269 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:34.930905 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbx7k" podUID="4a784083-f7eb-4508-9ef5-07ae8c51a973" Sep 07 03:48:35.601660 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:35.601618 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:35.601909 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:35.601769 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:35.601909 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:35.601846 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret podName:4a784083-f7eb-4508-9ef5-07ae8c51a973 nodeName:}" failed. No retries permitted until 2026-09-07 03:49:07.601825892 +0000 UTC m=+66.176692175 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret") pod "global-pull-secret-syncer-gbx7k" (UID: "4a784083-f7eb-4508-9ef5-07ae8c51a973") : object "kube-system"/"original-pull-secret" not registered Sep 07 03:48:35.701884 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:35.701852 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk4m\" (UniqueName: \"kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m\") pod \"network-check-target-x6689\" (UID: \"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6\") " pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:35.702080 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:35.701906 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:35.702080 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:35.702016 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:35.702080 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:35.702040 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 03:48:35.702080 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:35.702080 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 03:48:35.702280 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:35.702090 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs podName:9d176a6b-c22a-4f6a-9551-1dbd25928cf8 nodeName:}" failed. No retries permitted until 2026-09-07 03:49:07.702069622 +0000 UTC m=+66.276935906 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs") pod "network-metrics-daemon-sxp6n" (UID: "9d176a6b-c22a-4f6a-9551-1dbd25928cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 03:48:35.702280 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:35.702095 2555 projected.go:196] Error preparing data for projected volume kube-api-access-7zk4m for pod openshift-network-diagnostics/network-check-target-x6689: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:35.702280 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:35.702142 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m podName:1c70ff1e-fc11-4f58-80b5-98a9901fd7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:49:07.702126046 +0000 UTC m=+66.276992331 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zk4m" (UniqueName: "kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m") pod "network-check-target-x6689" (UID: "1c70ff1e-fc11-4f58-80b5-98a9901fd7d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 03:48:35.930304 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:35.930221 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:35.930476 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:35.930334 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6689" podUID="1c70ff1e-fc11-4f58-80b5-98a9901fd7d6" Sep 07 03:48:35.930476 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:35.930392 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:35.930476 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:35.930464 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxp6n" podUID="9d176a6b-c22a-4f6a-9551-1dbd25928cf8" Sep 07 03:48:36.680440 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.680255 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-215.ec2.internal" event="NodeReady" Sep 07 03:48:36.680832 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.680545 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 07 03:48:36.717779 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.717745 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zvrg9"] Sep 07 03:48:36.724081 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.723498 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7447dd5b74-p7pm5"] Sep 07 03:48:36.724081 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.723767 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.727294 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.727269 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 07 03:48:36.727483 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.727463 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 07 03:48:36.727578 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.727564 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 07 03:48:36.727814 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.727798 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c"] Sep 07 03:48:36.727922 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.727907 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:36.729101 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.729078 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wvlt9\"" Sep 07 03:48:36.729226 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.729139 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 07 03:48:36.733946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.731103 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 07 03:48:36.733946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.731534 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 07 03:48:36.733946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.731944 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 07 03:48:36.733946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.733759 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk"] Sep 07 03:48:36.733946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.733870 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c" Sep 07 03:48:36.735523 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.734741 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 07 03:48:36.735523 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.734800 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dgwxt\"" Sep 07 03:48:36.735523 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.735018 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 07 03:48:36.735523 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.735025 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 07 03:48:36.736421 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.736402 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6jlbj\"" Sep 07 03:48:36.736421 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.736418 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 07 03:48:36.736590 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.736484 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 07 03:48:36.739331 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.738495 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c"] Sep 07 03:48:36.739331 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.739016 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 07 03:48:36.739501 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.739430 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:36.741529 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.741511 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 07 03:48:36.741905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.741889 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 07 03:48:36.742027 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.742007 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gv6ft\"" Sep 07 03:48:36.742349 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.742330 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 07 03:48:36.746753 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.746382 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b7f8788ff-2zcb2"] Sep 07 03:48:36.746933 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.746858 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:36.749733 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.749709 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-bml8f"] Sep 07 03:48:36.749840 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.749823 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 07 03:48:36.749899 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.749851 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zwwvl\"" Sep 07 03:48:36.750049 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.750032 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 07 03:48:36.750167 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.750125 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 07 03:48:36.750223 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.750169 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:36.750338 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.750322 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 07 03:48:36.752302 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.752286 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 07 03:48:36.752613 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.752568 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 07 03:48:36.752791 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.752335 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-njlgh\"" Sep 07 03:48:36.753200 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.753182 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn"] Sep 07 03:48:36.753343 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.753327 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:36.753411 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.753199 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 07 03:48:36.755490 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.755472 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 07 03:48:36.755884 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.755867 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 07 03:48:36.755962 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.755890 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 07 03:48:36.756105 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.756048 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 07 03:48:36.756388 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.756372 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5zr7b\"" Sep 07 03:48:36.757306 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.757234 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s"] Sep 07 03:48:36.757543 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.757523 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:36.761635 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.759852 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 07 03:48:36.764952 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.763000 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 07 03:48:36.764952 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.763656 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ttsv2\"" Sep 07 03:48:36.764952 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.764459 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 07 03:48:36.765233 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.764963 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 07 03:48:36.765616 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.765598 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 07 03:48:36.767364 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.767310 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zvrg9"] Sep 07 03:48:36.767364 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.767350 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c"] Sep 07 03:48:36.767853 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.767371 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk"] Sep 07 03:48:36.767853 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.767402 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7447dd5b74-p7pm5"] Sep 07 03:48:36.767853 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.767420 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-bml8f"] Sep 07 03:48:36.767853 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.767444 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b7f8788ff-2zcb2"] Sep 07 03:48:36.767853 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.767466 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gljg7"] Sep 07 03:48:36.768169 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.768050 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:36.770777 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.770754 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 07 03:48:36.772120 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.772090 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bqnfb\"" Sep 07 03:48:36.772120 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.772104 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 07 03:48:36.772632 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.772614 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rh982"] Sep 07 03:48:36.773293 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.773274 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 07 03:48:36.773650 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.773631 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 07 03:48:36.773650 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.773859 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 07 03:48:36.774224 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.774131 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gljg7" Sep 07 03:48:36.776240 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.776220 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 07 03:48:36.776331 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.776275 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 07 03:48:36.776532 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.776516 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 07 03:48:36.776604 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.776572 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b5xmc\"" Sep 07 03:48:36.777950 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.777933 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn"] Sep 07 03:48:36.778029 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.777966 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-2srz4"] Sep 07 03:48:36.778087 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.778069 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rh982" Sep 07 03:48:36.780681 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.780663 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 07 03:48:36.781200 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.780906 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 07 03:48:36.781200 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.781152 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s"] Sep 07 03:48:36.781200 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.781184 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gljg7"] Sep 07 03:48:36.781200 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.781203 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-2srz4"] Sep 07 03:48:36.781432 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.781220 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rh982"] Sep 07 03:48:36.781432 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.781234 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-2srz4" Sep 07 03:48:36.781432 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.781243 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c"] Sep 07 03:48:36.781432 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.780941 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n9q8q\"" Sep 07 03:48:36.783448 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.783433 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vzb8j\"" Sep 07 03:48:36.783658 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.783642 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 03:48:36.783771 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.783755 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 03:48:36.809992 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.809973 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/577e74b9-769c-48fa-b70f-e87314e9f179-config\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.810115 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.810024 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/577e74b9-769c-48fa-b70f-e87314e9f179-serving-cert\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.810185 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.810123 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/577e74b9-769c-48fa-b70f-e87314e9f179-trusted-ca\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.810185 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.810167 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjdc4\" (UniqueName: \"kubernetes.io/projected/577e74b9-769c-48fa-b70f-e87314e9f179-kube-api-access-mjdc4\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.898522 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.898479 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:36.911138 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911108 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66017c83-f8c1-4280-aefa-499d9861a68f-ca-trust-extracted\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:36.911253 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911145 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l89l\" (UniqueName: \"kubernetes.io/projected/4a5acee8-01de-477c-a111-d0f3ea9b2afb-kube-api-access-8l89l\") pod \"service-ca-operator-859f94c4b-5m95s\" (UID: \"4a5acee8-01de-477c-a111-d0f3ea9b2afb\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:36.911253 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911173 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdb59\" (UniqueName: \"kubernetes.io/projected/1503ab01-93bb-4c30-95ec-e8c7fb37b620-kube-api-access-jdb59\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:36.911253 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911200 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68mbl\" (UniqueName: \"kubernetes.io/projected/8b8f75fd-b50c-461a-9f2f-d8433861d39a-kube-api-access-68mbl\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:36.911253 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911224 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/212b6720-24d5-46df-8053-6859f2e790ca-cert\") pod \"ingress-canary-gljg7\" (UID: \"212b6720-24d5-46df-8053-6859f2e790ca\") " pod="openshift-ingress-canary/ingress-canary-gljg7" Sep 07 03:48:36.911253 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911244 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5e0dcd-3686-4f2d-8710-36f47ade6997-serving-cert\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911263 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e43e9f6-5add-434b-89cc-d9db04f24dc7-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rmznn\" (UID: \"4e43e9f6-5add-434b-89cc-d9db04f24dc7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911296 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e43e9f6-5add-434b-89cc-d9db04f24dc7-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rmznn\" (UID: \"4e43e9f6-5add-434b-89cc-d9db04f24dc7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911330 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b8f75fd-b50c-461a-9f2f-d8433861d39a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911357 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwtqb\" (UniqueName: \"kubernetes.io/projected/cc36ab08-3dc2-4011-b24a-4ee2635c4630-kube-api-access-qwtqb\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911376 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-tmp-dir\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911401 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96g7\" (UniqueName: \"kubernetes.io/projected/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-kube-api-access-l96g7\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911419 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2c99\" (UniqueName: \"kubernetes.io/projected/4e43e9f6-5add-434b-89cc-d9db04f24dc7-kube-api-access-d2c99\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rmznn\" (UID: \"4e43e9f6-5add-434b-89cc-d9db04f24dc7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911448 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911470 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-bound-sa-token\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911502 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911530 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:36.911561 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911557 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-registry-certificates\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911594 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6drv\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-kube-api-access-z6drv\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911630 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a5acee8-01de-477c-a111-d0f3ea9b2afb-config\") pod \"service-ca-operator-859f94c4b-5m95s\" (UID: \"4a5acee8-01de-477c-a111-d0f3ea9b2afb\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911656 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jtq\" (UniqueName: \"kubernetes.io/projected/30ef3e57-2a2d-417d-8110-1f36a6bd910d-kube-api-access-q6jtq\") pod \"network-check-source-685456655f-2srz4\" (UID: \"30ef3e57-2a2d-417d-8110-1f36a6bd910d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-2srz4" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911677 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed5e0dcd-3686-4f2d-8710-36f47ade6997-tmp\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911710 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/577e74b9-769c-48fa-b70f-e87314e9f179-config\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911735 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-installation-pull-secrets\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911759 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/577e74b9-769c-48fa-b70f-e87314e9f179-serving-cert\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911791 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-config-volume\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911818 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75nzb\" (UniqueName: \"kubernetes.io/projected/6ea44a63-9e02-4390-b39a-d849c618285a-kube-api-access-75nzb\") pod \"volume-data-source-validator-695c895bd6-ltc2c\" (UID: \"6ea44a63-9e02-4390-b39a-d849c618285a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911855 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a5acee8-01de-477c-a111-d0f3ea9b2afb-serving-cert\") pod \"service-ca-operator-859f94c4b-5m95s\" (UID: \"4a5acee8-01de-477c-a111-d0f3ea9b2afb\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911897 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-stats-auth\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911923 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911941 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ed5e0dcd-3686-4f2d-8710-36f47ade6997-snapshots\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911960 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:36.912145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.911986 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/577e74b9-769c-48fa-b70f-e87314e9f179-trusted-ca\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912017 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-image-registry-private-configuration\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912101 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44w9\" (UniqueName: \"kubernetes.io/projected/212b6720-24d5-46df-8053-6859f2e790ca-kube-api-access-z44w9\") pod \"ingress-canary-gljg7\" (UID: \"212b6720-24d5-46df-8053-6859f2e790ca\") " pod="openshift-ingress-canary/ingress-canary-gljg7" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912127 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjdc4\" (UniqueName: \"kubernetes.io/projected/577e74b9-769c-48fa-b70f-e87314e9f179-kube-api-access-mjdc4\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912150 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-trusted-ca\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912174 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e0dcd-3686-4f2d-8710-36f47ade6997-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912210 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-default-certificate\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912251 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e0dcd-3686-4f2d-8710-36f47ade6997-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912281 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hfj\" (UniqueName: \"kubernetes.io/projected/ed5e0dcd-3686-4f2d-8710-36f47ade6997-kube-api-access-f4hfj\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912307 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912364 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/577e74b9-769c-48fa-b70f-e87314e9f179-config\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.912786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.912761 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/577e74b9-769c-48fa-b70f-e87314e9f179-trusted-ca\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.916266 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.916244 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/577e74b9-769c-48fa-b70f-e87314e9f179-serving-cert\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.923495 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.923475 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjdc4\" (UniqueName: \"kubernetes.io/projected/577e74b9-769c-48fa-b70f-e87314e9f179-kube-api-access-mjdc4\") pod \"console-operator-85b454874b-zvrg9\" (UID: \"577e74b9-769c-48fa-b70f-e87314e9f179\") " pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:36.930259 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.930240 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:48:36.933166 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.933146 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 03:48:36.989855 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:36.989818 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:37.012785 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.012728 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96g7\" (UniqueName: \"kubernetes.io/projected/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-kube-api-access-l96g7\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:37.012785 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.012756 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2c99\" (UniqueName: \"kubernetes.io/projected/4e43e9f6-5add-434b-89cc-d9db04f24dc7-kube-api-access-d2c99\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rmznn\" (UID: \"4e43e9f6-5add-434b-89cc-d9db04f24dc7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:37.012946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.012790 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:37.012946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.012813 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-bound-sa-token\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.012946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.012835 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:37.012946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.012856 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.012946 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.012882 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-registry-certificates\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.013149 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.012951 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:37.013149 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.013025 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls podName:8b8f75fd-b50c-461a-9f2f-d8433861d39a nodeName:}" failed. No retries permitted until 2026-09-07 03:48:37.513005093 +0000 UTC m=+36.087871365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5qh6c" (UID: "8b8f75fd-b50c-461a-9f2f-d8433861d39a") : secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:37.013149 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013117 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6drv\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-kube-api-access-z6drv\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.013149 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013143 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a5acee8-01de-477c-a111-d0f3ea9b2afb-config\") pod \"service-ca-operator-859f94c4b-5m95s\" (UID: \"4a5acee8-01de-477c-a111-d0f3ea9b2afb\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:37.013309 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013161 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jtq\" (UniqueName: \"kubernetes.io/projected/30ef3e57-2a2d-417d-8110-1f36a6bd910d-kube-api-access-q6jtq\") pod \"network-check-source-685456655f-2srz4\" (UID: \"30ef3e57-2a2d-417d-8110-1f36a6bd910d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-2srz4" Sep 07 03:48:37.013309 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013177 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed5e0dcd-3686-4f2d-8710-36f47ade6997-tmp\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.013309 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013197 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-installation-pull-secrets\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.013309 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013230 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-config-volume\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:37.013309 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013245 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75nzb\" (UniqueName: \"kubernetes.io/projected/6ea44a63-9e02-4390-b39a-d849c618285a-kube-api-access-75nzb\") pod \"volume-data-source-validator-695c895bd6-ltc2c\" (UID: \"6ea44a63-9e02-4390-b39a-d849c618285a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c" Sep 07 03:48:37.013309 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013263 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a5acee8-01de-477c-a111-d0f3ea9b2afb-serving-cert\") pod \"service-ca-operator-859f94c4b-5m95s\" (UID: \"4a5acee8-01de-477c-a111-d0f3ea9b2afb\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:37.013309 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013279 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-stats-auth\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:37.013309 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013296 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:37.013309 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013312 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ed5e0dcd-3686-4f2d-8710-36f47ade6997-snapshots\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.013698 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013330 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:37.013698 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013348 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-image-registry-private-configuration\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.013698 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013373 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44w9\" (UniqueName: \"kubernetes.io/projected/212b6720-24d5-46df-8053-6859f2e790ca-kube-api-access-z44w9\") pod \"ingress-canary-gljg7\" (UID: \"212b6720-24d5-46df-8053-6859f2e790ca\") " pod="openshift-ingress-canary/ingress-canary-gljg7" Sep 07 03:48:37.013698 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013403 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-trusted-ca\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.013698 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013427 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e0dcd-3686-4f2d-8710-36f47ade6997-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.013698 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.013462 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 03:48:37.013698 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.013484 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7f8788ff-2zcb2: secret "image-registry-tls" not found Sep 07 03:48:37.013698 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.013539 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls podName:66017c83-f8c1-4280-aefa-499d9861a68f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:37.513523004 +0000 UTC m=+36.088389285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls") pod "image-registry-7b7f8788ff-2zcb2" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f") : secret "image-registry-tls" not found Sep 07 03:48:37.013698 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.013467 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-default-certificate\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:37.014222 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014049 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e0dcd-3686-4f2d-8710-36f47ade6997-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.014222 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014120 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hfj\" (UniqueName: \"kubernetes.io/projected/ed5e0dcd-3686-4f2d-8710-36f47ade6997-kube-api-access-f4hfj\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.014222 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014160 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:37.014222 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014208 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66017c83-f8c1-4280-aefa-499d9861a68f-ca-trust-extracted\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.014459 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014239 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l89l\" (UniqueName: \"kubernetes.io/projected/4a5acee8-01de-477c-a111-d0f3ea9b2afb-kube-api-access-8l89l\") pod \"service-ca-operator-859f94c4b-5m95s\" (UID: \"4a5acee8-01de-477c-a111-d0f3ea9b2afb\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:37.014459 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014285 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdb59\" (UniqueName: \"kubernetes.io/projected/1503ab01-93bb-4c30-95ec-e8c7fb37b620-kube-api-access-jdb59\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:37.014459 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014313 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68mbl\" (UniqueName: \"kubernetes.io/projected/8b8f75fd-b50c-461a-9f2f-d8433861d39a-kube-api-access-68mbl\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:37.014459 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014357 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/212b6720-24d5-46df-8053-6859f2e790ca-cert\") pod \"ingress-canary-gljg7\" (UID: \"212b6720-24d5-46df-8053-6859f2e790ca\") " pod="openshift-ingress-canary/ingress-canary-gljg7" Sep 07 03:48:37.014459 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014382 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5e0dcd-3686-4f2d-8710-36f47ade6997-serving-cert\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.014459 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014420 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e43e9f6-5add-434b-89cc-d9db04f24dc7-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rmznn\" (UID: \"4e43e9f6-5add-434b-89cc-d9db04f24dc7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:37.014778 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014469 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-config-volume\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:37.014778 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014489 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e43e9f6-5add-434b-89cc-d9db04f24dc7-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rmznn\" (UID: \"4e43e9f6-5add-434b-89cc-d9db04f24dc7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:37.014778 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014520 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b8f75fd-b50c-461a-9f2f-d8433861d39a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:37.014928 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.014791 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-registry-certificates\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.014928 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.014850 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 03:48:37.014928 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.014910 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls podName:08336dd4-dd4c-47e3-ba8d-531ca667a0b5 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:37.514893592 +0000 UTC m=+36.089759861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls") pod "dns-default-rh982" (UID: "08336dd4-dd4c-47e3-ba8d-531ca667a0b5") : secret "dns-default-metrics-tls" not found Sep 07 03:48:37.015218 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.015192 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a5acee8-01de-477c-a111-d0f3ea9b2afb-config\") pod \"service-ca-operator-859f94c4b-5m95s\" (UID: \"4a5acee8-01de-477c-a111-d0f3ea9b2afb\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:37.015218 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.015212 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b8f75fd-b50c-461a-9f2f-d8433861d39a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:37.015377 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.015244 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:37.515224954 +0000 UTC m=+36.090091224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : configmap references non-existent config key: service-ca.crt Sep 07 03:48:37.015377 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.015317 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 03:48:37.015377 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.015371 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls podName:1503ab01-93bb-4c30-95ec-e8c7fb37b620 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:37.515353682 +0000 UTC m=+36.090219966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2n4bk" (UID: "1503ab01-93bb-4c30-95ec-e8c7fb37b620") : secret "samples-operator-tls" not found Sep 07 03:48:37.018913 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.015563 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 03:48:37.018913 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.015623 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:37.515608534 +0000 UTC m=+36.090474820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : secret "router-metrics-certs-default" not found Sep 07 03:48:37.018913 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.015872 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e0dcd-3686-4f2d-8710-36f47ade6997-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.018913 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.015903 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66017c83-f8c1-4280-aefa-499d9861a68f-ca-trust-extracted\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.018913 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.016164 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ed5e0dcd-3686-4f2d-8710-36f47ade6997-snapshots\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.018913 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.017478 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-trusted-ca\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.019230 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.018929 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e0dcd-3686-4f2d-8710-36f47ade6997-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.019230 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.019206 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed5e0dcd-3686-4f2d-8710-36f47ade6997-tmp\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.019372 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.019320 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e43e9f6-5add-434b-89cc-d9db04f24dc7-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rmznn\" (UID: \"4e43e9f6-5add-434b-89cc-d9db04f24dc7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:37.019372 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.019349 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/212b6720-24d5-46df-8053-6859f2e790ca-cert\") pod \"ingress-canary-gljg7\" (UID: \"212b6720-24d5-46df-8053-6859f2e790ca\") " pod="openshift-ingress-canary/ingress-canary-gljg7" Sep 07 03:48:37.019754 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.019697 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwtqb\" (UniqueName: \"kubernetes.io/projected/cc36ab08-3dc2-4011-b24a-4ee2635c4630-kube-api-access-qwtqb\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:37.019827 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.019738 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-tmp-dir\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:37.020316 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.020292 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-tmp-dir\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:37.020722 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.020699 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-default-certificate\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:37.021871 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.021846 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e43e9f6-5add-434b-89cc-d9db04f24dc7-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rmznn\" (UID: \"4e43e9f6-5add-434b-89cc-d9db04f24dc7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:37.023769 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.023751 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5e0dcd-3686-4f2d-8710-36f47ade6997-serving-cert\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.024457 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.024123 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-installation-pull-secrets\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.024457 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.024412 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-image-registry-private-configuration\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.025966 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.025943 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a5acee8-01de-477c-a111-d0f3ea9b2afb-serving-cert\") pod \"service-ca-operator-859f94c4b-5m95s\" (UID: \"4a5acee8-01de-477c-a111-d0f3ea9b2afb\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:37.029625 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.029328 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-stats-auth\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:37.029625 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.029553 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2c99\" (UniqueName: \"kubernetes.io/projected/4e43e9f6-5add-434b-89cc-d9db04f24dc7-kube-api-access-d2c99\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rmznn\" (UID: \"4e43e9f6-5add-434b-89cc-d9db04f24dc7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:37.029625 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.029554 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96g7\" (UniqueName: \"kubernetes.io/projected/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-kube-api-access-l96g7\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:37.029913 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.029895 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6drv\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-kube-api-access-z6drv\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.029978 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.029923 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l89l\" (UniqueName: \"kubernetes.io/projected/4a5acee8-01de-477c-a111-d0f3ea9b2afb-kube-api-access-8l89l\") pod \"service-ca-operator-859f94c4b-5m95s\" (UID: \"4a5acee8-01de-477c-a111-d0f3ea9b2afb\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:37.030215 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.030191 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jtq\" (UniqueName: \"kubernetes.io/projected/30ef3e57-2a2d-417d-8110-1f36a6bd910d-kube-api-access-q6jtq\") pod \"network-check-source-685456655f-2srz4\" (UID: \"30ef3e57-2a2d-417d-8110-1f36a6bd910d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-2srz4" Sep 07 03:48:37.030356 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.030301 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hfj\" (UniqueName: \"kubernetes.io/projected/ed5e0dcd-3686-4f2d-8710-36f47ade6997-kube-api-access-f4hfj\") pod \"insights-operator-76bcd6ccf5-bml8f\" (UID: \"ed5e0dcd-3686-4f2d-8710-36f47ade6997\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.030483 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.030373 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-bound-sa-token\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.030483 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.030474 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdb59\" (UniqueName: \"kubernetes.io/projected/1503ab01-93bb-4c30-95ec-e8c7fb37b620-kube-api-access-jdb59\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:37.030953 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.030934 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44w9\" (UniqueName: \"kubernetes.io/projected/212b6720-24d5-46df-8053-6859f2e790ca-kube-api-access-z44w9\") pod \"ingress-canary-gljg7\" (UID: \"212b6720-24d5-46df-8053-6859f2e790ca\") " pod="openshift-ingress-canary/ingress-canary-gljg7" Sep 07 03:48:37.031228 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.031211 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75nzb\" (UniqueName: \"kubernetes.io/projected/6ea44a63-9e02-4390-b39a-d849c618285a-kube-api-access-75nzb\") pod \"volume-data-source-validator-695c895bd6-ltc2c\" (UID: \"6ea44a63-9e02-4390-b39a-d849c618285a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c" Sep 07 03:48:37.032812 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.032771 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68mbl\" (UniqueName: \"kubernetes.io/projected/8b8f75fd-b50c-461a-9f2f-d8433861d39a-kube-api-access-68mbl\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:37.033509 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.033431 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwtqb\" (UniqueName: \"kubernetes.io/projected/cc36ab08-3dc2-4011-b24a-4ee2635c4630-kube-api-access-qwtqb\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:37.040534 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.040516 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:37.057411 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.057394 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c" Sep 07 03:48:37.090631 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.090611 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" Sep 07 03:48:37.096458 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.096441 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" Sep 07 03:48:37.103201 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.103181 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" Sep 07 03:48:37.109879 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.109863 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gljg7" Sep 07 03:48:37.121068 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.121031 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-2srz4" Sep 07 03:48:37.524855 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.524818 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:37.525038 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.524866 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:37.525038 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.524927 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:37.525038 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.524963 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 03:48:37.525038 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.524975 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:37.525038 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.525011 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:37.525038 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.525022 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls podName:08336dd4-dd4c-47e3-ba8d-531ca667a0b5 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:38.525004409 +0000 UTC m=+37.099870682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls") pod "dns-default-rh982" (UID: "08336dd4-dd4c-47e3-ba8d-531ca667a0b5") : secret "dns-default-metrics-tls" not found Sep 07 03:48:37.525372 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.525051 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 03:48:37.525372 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.525083 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7f8788ff-2zcb2: secret "image-registry-tls" not found Sep 07 03:48:37.525372 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.525090 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls podName:8b8f75fd-b50c-461a-9f2f-d8433861d39a nodeName:}" failed. No retries permitted until 2026-09-07 03:48:38.525074937 +0000 UTC m=+37.099941206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5qh6c" (UID: "8b8f75fd-b50c-461a-9f2f-d8433861d39a") : secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:37.525372 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.525108 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 03:48:37.525372 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.525130 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:37.525372 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.525149 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls podName:66017c83-f8c1-4280-aefa-499d9861a68f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:38.525131536 +0000 UTC m=+37.099997805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls") pod "image-registry-7b7f8788ff-2zcb2" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f") : secret "image-registry-tls" not found Sep 07 03:48:37.525372 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.525168 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls podName:1503ab01-93bb-4c30-95ec-e8c7fb37b620 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:38.525162009 +0000 UTC m=+37.100028276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2n4bk" (UID: "1503ab01-93bb-4c30-95ec-e8c7fb37b620") : secret "samples-operator-tls" not found Sep 07 03:48:37.525372 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.525217 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:38.525206792 +0000 UTC m=+37.100073086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : configmap references non-existent config key: service-ca.crt Sep 07 03:48:37.525372 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.525244 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:37.525372 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.525365 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 03:48:37.525683 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:37.525422 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:38.525388789 +0000 UTC m=+37.100255097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : secret "router-metrics-certs-default" not found Sep 07 03:48:37.935253 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.931373 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:48:37.935253 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.932328 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:48:37.938977 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.938359 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wrs9m\"" Sep 07 03:48:37.938977 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.938614 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 03:48:37.938977 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:37.938789 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-25ldn\"" Sep 07 03:48:38.231859 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.231834 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s"] Sep 07 03:48:38.236238 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:38.236215 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5acee8_01de_477c_a111_d0f3ea9b2afb.slice/crio-b7da7ad92f063479205056a19668a4a7acb29821dd57248e0872ff566a351ba5 WatchSource:0}: Error finding container b7da7ad92f063479205056a19668a4a7acb29821dd57248e0872ff566a351ba5: Status 404 returned error can't find the container with id b7da7ad92f063479205056a19668a4a7acb29821dd57248e0872ff566a351ba5 Sep 07 03:48:38.241339 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.241193 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zvrg9"] Sep 07 03:48:38.244675 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.244650 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c"] Sep 07 03:48:38.245401 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:38.245380 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577e74b9_769c_48fa_b70f_e87314e9f179.slice/crio-ca129aa36ec8b8da9e1597ed75b27a49676ac0b887a6adb1527f48bbae0a94ef WatchSource:0}: Error finding container ca129aa36ec8b8da9e1597ed75b27a49676ac0b887a6adb1527f48bbae0a94ef: Status 404 returned error can't find the container with id ca129aa36ec8b8da9e1597ed75b27a49676ac0b887a6adb1527f48bbae0a94ef Sep 07 03:48:38.246146 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.245980 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-2srz4"] Sep 07 03:48:38.250382 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:38.249894 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ef3e57_2a2d_417d_8110_1f36a6bd910d.slice/crio-c9da50077be2633f3b5082335897b60e8f3fbe2dcb2504b6ba372e55659e74ad WatchSource:0}: Error finding container c9da50077be2633f3b5082335897b60e8f3fbe2dcb2504b6ba372e55659e74ad: Status 404 returned error can't find the container with id c9da50077be2633f3b5082335897b60e8f3fbe2dcb2504b6ba372e55659e74ad Sep 07 03:48:38.250824 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:38.250803 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea44a63_9e02_4390_b39a_d849c618285a.slice/crio-637a04379cbeabe2840e02f26d558c8923e812dce93f6dd3310a6c7c0d31a3cd WatchSource:0}: Error finding container 637a04379cbeabe2840e02f26d558c8923e812dce93f6dd3310a6c7c0d31a3cd: Status 404 returned error can't find the container with id 637a04379cbeabe2840e02f26d558c8923e812dce93f6dd3310a6c7c0d31a3cd Sep 07 03:48:38.252148 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.252082 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-bml8f"] Sep 07 03:48:38.253618 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.253453 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gljg7"] Sep 07 03:48:38.255128 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.254646 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn"] Sep 07 03:48:38.536970 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.536944 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:38.537097 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.536977 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:38.537097 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.536995 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:38.537097 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.537046 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.537106 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537129 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537144 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537160 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7f8788ff-2zcb2: secret "image-registry-tls" not found Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:38.537162 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537133 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537204 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls podName:8b8f75fd-b50c-461a-9f2f-d8433861d39a nodeName:}" failed. No retries permitted until 2026-09-07 03:48:40.537183388 +0000 UTC m=+39.112049661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5qh6c" (UID: "8b8f75fd-b50c-461a-9f2f-d8433861d39a") : secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537210 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537221 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls podName:66017c83-f8c1-4280-aefa-499d9861a68f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:40.537212399 +0000 UTC m=+39.112078668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls") pod "image-registry-7b7f8788ff-2zcb2" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f") : secret "image-registry-tls" not found Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537236 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls podName:08336dd4-dd4c-47e3-ba8d-531ca667a0b5 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:40.537228231 +0000 UTC m=+39.112094502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls") pod "dns-default-rh982" (UID: "08336dd4-dd4c-47e3-ba8d-531ca667a0b5") : secret "dns-default-metrics-tls" not found Sep 07 03:48:38.537257 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537265 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls podName:1503ab01-93bb-4c30-95ec-e8c7fb37b620 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:40.537245423 +0000 UTC m=+39.112111694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2n4bk" (UID: "1503ab01-93bb-4c30-95ec-e8c7fb37b620") : secret "samples-operator-tls" not found Sep 07 03:48:38.537625 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537276 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 03:48:38.537625 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537311 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:40.537299255 +0000 UTC m=+39.112165525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : configmap references non-existent config key: service-ca.crt Sep 07 03:48:38.537625 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:38.537326 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:40.537317902 +0000 UTC m=+39.112184175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : secret "router-metrics-certs-default" not found Sep 07 03:48:39.200993 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.200909 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" event={"ID":"4e43e9f6-5add-434b-89cc-d9db04f24dc7","Type":"ContainerStarted","Data":"41884df53dc4e9a199cced67a9943380885f82d9bffad7bed7e8754f0b2017dc"} Sep 07 03:48:39.204710 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.204648 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-2srz4" event={"ID":"30ef3e57-2a2d-417d-8110-1f36a6bd910d","Type":"ContainerStarted","Data":"c9da50077be2633f3b5082335897b60e8f3fbe2dcb2504b6ba372e55659e74ad"} Sep 07 03:48:39.206165 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.206115 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" event={"ID":"4a5acee8-01de-477c-a111-d0f3ea9b2afb","Type":"ContainerStarted","Data":"b7da7ad92f063479205056a19668a4a7acb29821dd57248e0872ff566a351ba5"} Sep 07 03:48:39.209928 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.209859 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gljg7" event={"ID":"212b6720-24d5-46df-8053-6859f2e790ca","Type":"ContainerStarted","Data":"d9b8bdcc2444477d72490782f192ea7616f4a248995e5fa41ee505c2deb41802"} Sep 07 03:48:39.213016 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.212949 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" event={"ID":"577e74b9-769c-48fa-b70f-e87314e9f179","Type":"ContainerStarted","Data":"ca129aa36ec8b8da9e1597ed75b27a49676ac0b887a6adb1527f48bbae0a94ef"} Sep 07 03:48:39.217002 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.216973 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" event={"ID":"ed5e0dcd-3686-4f2d-8710-36f47ade6997","Type":"ContainerStarted","Data":"e9b4e3e2bee69b64e6f1e592552f1e267cfc52e6c12c1b3008489e7494bea56d"} Sep 07 03:48:39.220696 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.220672 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c" event={"ID":"6ea44a63-9e02-4390-b39a-d849c618285a","Type":"ContainerStarted","Data":"637a04379cbeabe2840e02f26d558c8923e812dce93f6dd3310a6c7c0d31a3cd"} Sep 07 03:48:39.228168 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.227300 2555 generic.go:356] "Generic (PLEG): container finished" podID="1231ac90-f33a-4adb-bfb4-8bd0b14c9348" containerID="dc058e724de7132aee66b307c5cdef3c92bcc36f9a7e2e8f98f49fc80d1810ba" exitCode=0 Sep 07 03:48:39.228168 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.227337 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" event={"ID":"1231ac90-f33a-4adb-bfb4-8bd0b14c9348","Type":"ContainerDied","Data":"dc058e724de7132aee66b307c5cdef3c92bcc36f9a7e2e8f98f49fc80d1810ba"} Sep 07 03:48:39.553820 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.553723 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:39.653224 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:39.653125 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:40.249805 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:40.249768 2555 generic.go:356] "Generic (PLEG): container finished" podID="1231ac90-f33a-4adb-bfb4-8bd0b14c9348" containerID="c46f8be03811a54ae31f9e60a53e3f24c107dd02ffbc9c17024bd522ba4706b4" exitCode=0 Sep 07 03:48:40.250378 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:40.249899 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" event={"ID":"1231ac90-f33a-4adb-bfb4-8bd0b14c9348","Type":"ContainerDied","Data":"c46f8be03811a54ae31f9e60a53e3f24c107dd02ffbc9c17024bd522ba4706b4"} Sep 07 03:48:40.515793 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:40.515689 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:40.557587 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:40.557624 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:40.557661 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:40.557712 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:40.557729 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:40.557746 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.557860 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.557871 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7f8788ff-2zcb2: secret "image-registry-tls" not found Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.557919 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls podName:66017c83-f8c1-4280-aefa-499d9861a68f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:44.557901022 +0000 UTC m=+43.132767306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls") pod "image-registry-7b7f8788ff-2zcb2" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f") : secret "image-registry-tls" not found Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.558302 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.558350 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls podName:1503ab01-93bb-4c30-95ec-e8c7fb37b620 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:44.558333776 +0000 UTC m=+43.133200060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2n4bk" (UID: "1503ab01-93bb-4c30-95ec-e8c7fb37b620") : secret "samples-operator-tls" not found Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.558415 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:44.558404725 +0000 UTC m=+43.133271012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : configmap references non-existent config key: service-ca.crt Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.558470 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.558492 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:44.558485306 +0000 UTC m=+43.133351573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : secret "router-metrics-certs-default" not found Sep 07 03:48:40.558670 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.558527 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:40.559269 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.558547 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls podName:8b8f75fd-b50c-461a-9f2f-d8433861d39a nodeName:}" failed. No retries permitted until 2026-09-07 03:48:44.558540922 +0000 UTC m=+43.133407189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5qh6c" (UID: "8b8f75fd-b50c-461a-9f2f-d8433861d39a") : secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:40.559269 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.558591 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 03:48:40.559269 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:40.558612 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls podName:08336dd4-dd4c-47e3-ba8d-531ca667a0b5 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:44.55860646 +0000 UTC m=+43.133472727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls") pod "dns-default-rh982" (UID: "08336dd4-dd4c-47e3-ba8d-531ca667a0b5") : secret "dns-default-metrics-tls" not found Sep 07 03:48:40.615791 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:40.615764 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:41.260127 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:41.259345 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" event={"ID":"1231ac90-f33a-4adb-bfb4-8bd0b14c9348","Type":"ContainerStarted","Data":"0c0b01fa6053ffab57f31f5168a9da25807dcc3751219f19815d0b0886572f35"} Sep 07 03:48:41.267546 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:41.267499 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:41.283736 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:41.282747 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wcg5f" podStartSLOduration=9.660120265 podStartE2EDuration="39.282725807s" totalImagesPullingTime="29.622605542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:41.280779096 +0000 UTC m=+39.855645386" watchObservedRunningTime="2026-09-07 03:48:41.282725807 +0000 UTC m=+39.857592097" Sep 07 03:48:41.369111 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:41.369069 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:41.468416 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:41.468385 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:41.529447 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:41.529359 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:43.461646 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:43.461608 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:44.388145 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:44.388106 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:44.460905 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:44.460850 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:44.594269 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:44.594226 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:44.594282 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:44.594333 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594398 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:44.594410 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:44.594450 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594477 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls podName:1503ab01-93bb-4c30-95ec-e8c7fb37b620 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:52.594456103 +0000 UTC m=+51.169322389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2n4bk" (UID: "1503ab01-93bb-4c30-95ec-e8c7fb37b620") : secret "samples-operator-tls" not found Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:44.594516 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594521 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594551 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594545 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:52.594535292 +0000 UTC m=+51.169401560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : configmap references non-existent config key: service-ca.crt Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594581 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594597 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls podName:08336dd4-dd4c-47e3-ba8d-531ca667a0b5 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:52.594584811 +0000 UTC m=+51.169451084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls") pod "dns-default-rh982" (UID: "08336dd4-dd4c-47e3-ba8d-531ca667a0b5") : secret "dns-default-metrics-tls" not found Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594614 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:52.594609355 +0000 UTC m=+51.169475621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : secret "router-metrics-certs-default" not found Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594623 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594648 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7f8788ff-2zcb2: secret "image-registry-tls" not found Sep 07 03:48:44.594690 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594627 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls podName:8b8f75fd-b50c-461a-9f2f-d8433861d39a nodeName:}" failed. No retries permitted until 2026-09-07 03:48:52.594622652 +0000 UTC m=+51.169488919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5qh6c" (UID: "8b8f75fd-b50c-461a-9f2f-d8433861d39a") : secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:44.595382 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:44.594708 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls podName:66017c83-f8c1-4280-aefa-499d9861a68f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:52.594692137 +0000 UTC m=+51.169558412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls") pod "image-registry-7b7f8788ff-2zcb2" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f") : secret "image-registry-tls" not found Sep 07 03:48:46.281624 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.280684 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" event={"ID":"ed5e0dcd-3686-4f2d-8710-36f47ade6997","Type":"ContainerStarted","Data":"e9f4381877e0651fb213a6a515573bf50e512a700b160fa617f3556364bf25c1"} Sep 07 03:48:46.287504 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.286738 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c" event={"ID":"6ea44a63-9e02-4390-b39a-d849c618285a","Type":"ContainerStarted","Data":"7eb16393473acddb01581eb9ca938a2adcff58a8e9554df21aeb9a0028017d39"} Sep 07 03:48:46.289116 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.289092 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" event={"ID":"4e43e9f6-5add-434b-89cc-d9db04f24dc7","Type":"ContainerStarted","Data":"87f75c27cc448b4fc879e342100c58d1bdd427bb62e9fb70ad847ac7719c418a"} Sep 07 03:48:46.291689 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.291636 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-2srz4" event={"ID":"30ef3e57-2a2d-417d-8110-1f36a6bd910d","Type":"ContainerStarted","Data":"4dcfffa71e2c76eba44593634d09339381483e520871e3609d06e79beef4ffc3"} Sep 07 03:48:46.293415 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.293395 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" event={"ID":"4a5acee8-01de-477c-a111-d0f3ea9b2afb","Type":"ContainerStarted","Data":"d99a70c3b6966c67411d71ae60d7b119f4bd27780730dcc86acd2c5c4a6f625a"} Sep 07 03:48:46.302179 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.301050 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" podStartSLOduration=29.492424708 podStartE2EDuration="37.301033278s" totalImagesPullingTime="7.80860857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:46.300666984 +0000 UTC m=+44.875533286" watchObservedRunningTime="2026-09-07 03:48:46.301033278 +0000 UTC m=+44.875899568" Sep 07 03:48:46.321209 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.321149 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" podStartSLOduration=19.508643696 podStartE2EDuration="27.321126714s" totalImagesPullingTime="7.812483018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:46.32016676 +0000 UTC m=+44.895033062" watchObservedRunningTime="2026-09-07 03:48:46.321126714 +0000 UTC m=+44.895993008" Sep 07 03:48:46.343680 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.342280 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-2srz4" podStartSLOduration=17.526905683 podStartE2EDuration="25.34226541s" totalImagesPullingTime="7.815359727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:46.34119934 +0000 UTC m=+44.916065631" watchObservedRunningTime="2026-09-07 03:48:46.34226541 +0000 UTC m=+44.917131697" Sep 07 03:48:46.379786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.379117 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ltc2c" podStartSLOduration=23.641570652 podStartE2EDuration="31.379096308s" totalImagesPullingTime="7.737525656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:46.359235983 +0000 UTC m=+44.934102269" watchObservedRunningTime="2026-09-07 03:48:46.379096308 +0000 UTC m=+44.953962593" Sep 07 03:48:46.379786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.379611 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" podStartSLOduration=19.551377733 podStartE2EDuration="27.379602118s" totalImagesPullingTime="7.828224385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:46.378890034 +0000 UTC m=+44.953756324" watchObservedRunningTime="2026-09-07 03:48:46.379602118 +0000 UTC m=+44.954468409" Sep 07 03:48:46.443727 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.443685 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:46.505602 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:46.505552 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:47.090681 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.090646 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:47.145743 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.145704 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:47.299228 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.299189 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gljg7" event={"ID":"212b6720-24d5-46df-8053-6859f2e790ca","Type":"ContainerStarted","Data":"827b69e11c6aaf52977a994ce1386c01136b81b054386bd6c6d80e45f7264f3e"} Sep 07 03:48:47.300904 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.300881 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/0.log" Sep 07 03:48:47.301015 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.300923 2555 generic.go:356] "Generic (PLEG): container finished" podID="577e74b9-769c-48fa-b70f-e87314e9f179" containerID="e13ecd3ad070993f611e095b78d47e51f0cc24c00560acaa2b969adbc1cae176" exitCode=255 Sep 07 03:48:47.301085 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.301043 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" event={"ID":"577e74b9-769c-48fa-b70f-e87314e9f179","Type":"ContainerDied","Data":"e13ecd3ad070993f611e095b78d47e51f0cc24c00560acaa2b969adbc1cae176"} Sep 07 03:48:47.301198 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.301182 2555 scope.go:117] "RemoveContainer" containerID="e13ecd3ad070993f611e095b78d47e51f0cc24c00560acaa2b969adbc1cae176" Sep 07 03:48:47.314934 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.314886 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gljg7" podStartSLOduration=3.49746883 podStartE2EDuration="11.314873161s" totalImagesPullingTime="7.817404331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:47.313954036 +0000 UTC m=+45.888820324" watchObservedRunningTime="2026-09-07 03:48:47.314873161 +0000 UTC m=+45.889739450" Sep 07 03:48:47.408089 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.405098 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wv9z9"] Sep 07 03:48:47.421676 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.421641 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wv9z9"] Sep 07 03:48:47.421843 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.421767 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.425797 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.425614 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 07 03:48:47.425797 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.425795 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 07 03:48:47.425973 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.425935 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7k4qm\"" Sep 07 03:48:47.462798 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.462769 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:47.524916 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.524888 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgcv\" (UniqueName: \"kubernetes.io/projected/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-api-access-ghgcv\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.525025 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.524921 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd7656b6-1017-4966-92af-fd1d32d08a7f-data-volume\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.525105 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.525082 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd7656b6-1017-4966-92af-fd1d32d08a7f-crio-socket\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.525207 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.525188 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.525267 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.525226 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.528625 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.528596 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:47.625652 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.625594 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgcv\" (UniqueName: \"kubernetes.io/projected/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-api-access-ghgcv\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.625652 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.625629 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd7656b6-1017-4966-92af-fd1d32d08a7f-data-volume\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.625818 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.625797 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd7656b6-1017-4966-92af-fd1d32d08a7f-crio-socket\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.625890 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.625873 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.625942 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.625900 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.625942 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.625911 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd7656b6-1017-4966-92af-fd1d32d08a7f-crio-socket\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.626019 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.625946 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd7656b6-1017-4966-92af-fd1d32d08a7f-data-volume\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.626019 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:47.625964 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 03:48:47.626019 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:47.625994 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 03:48:47.626178 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:47.626026 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm podName:dd7656b6-1017-4966-92af-fd1d32d08a7f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:48.126007511 +0000 UTC m=+46.700873784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wv9z9" (UID: "dd7656b6-1017-4966-92af-fd1d32d08a7f") : configmap "kube-rbac-proxy" not found Sep 07 03:48:47.626178 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:47.626044 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls podName:dd7656b6-1017-4966-92af-fd1d32d08a7f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:48.126033762 +0000 UTC m=+46.700900035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wv9z9" (UID: "dd7656b6-1017-4966-92af-fd1d32d08a7f") : secret "insights-runtime-extractor-tls" not found Sep 07 03:48:47.629767 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.629746 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:47.636826 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.636807 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgcv\" (UniqueName: \"kubernetes.io/projected/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-api-access-ghgcv\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:47.798247 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.798215 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:47.900845 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:47.900768 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:48.013223 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.013193 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:48.034206 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.034182 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t"] Sep 07 03:48:48.066012 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.065992 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t"] Sep 07 03:48:48.066117 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.066102 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t" Sep 07 03:48:48.069261 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.069232 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-l4l2q\"" Sep 07 03:48:48.069353 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.069274 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 07 03:48:48.069353 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.069316 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 07 03:48:48.084603 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.084580 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:48.131332 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.131313 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:48.131425 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.131342 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:48.131425 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.131388 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2ns\" (UniqueName: \"kubernetes.io/projected/10582104-f53f-40dd-9f07-81d404420cd4-kube-api-access-6h2ns\") pod \"migrator-fdbb8fdff-vnd5t\" (UID: \"10582104-f53f-40dd-9f07-81d404420cd4\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t" Sep 07 03:48:48.131537 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:48.131442 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 03:48:48.131537 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:48.131453 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 03:48:48.131537 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:48.131500 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls podName:dd7656b6-1017-4966-92af-fd1d32d08a7f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:49.131486986 +0000 UTC m=+47.706353253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wv9z9" (UID: "dd7656b6-1017-4966-92af-fd1d32d08a7f") : secret "insights-runtime-extractor-tls" not found Sep 07 03:48:48.131537 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:48.131514 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm podName:dd7656b6-1017-4966-92af-fd1d32d08a7f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:49.131507589 +0000 UTC m=+47.706373857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wv9z9" (UID: "dd7656b6-1017-4966-92af-fd1d32d08a7f") : configmap "kube-rbac-proxy" not found Sep 07 03:48:48.136675 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.136648 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:48.232534 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.232512 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2ns\" (UniqueName: \"kubernetes.io/projected/10582104-f53f-40dd-9f07-81d404420cd4-kube-api-access-6h2ns\") pod \"migrator-fdbb8fdff-vnd5t\" (UID: \"10582104-f53f-40dd-9f07-81d404420cd4\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t" Sep 07 03:48:48.240008 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.239981 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2ns\" (UniqueName: \"kubernetes.io/projected/10582104-f53f-40dd-9f07-81d404420cd4-kube-api-access-6h2ns\") pod \"migrator-fdbb8fdff-vnd5t\" (UID: \"10582104-f53f-40dd-9f07-81d404420cd4\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t" Sep 07 03:48:48.309048 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.309027 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 03:48:48.309476 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.309463 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/0.log" Sep 07 03:48:48.309535 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.309495 2555 generic.go:356] "Generic (PLEG): container finished" podID="577e74b9-769c-48fa-b70f-e87314e9f179" containerID="329d8138d4635f4a559e3331f5633e9c9d3d847e6c40de9596546311c934be6f" exitCode=255 Sep 07 03:48:48.309613 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.309592 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" event={"ID":"577e74b9-769c-48fa-b70f-e87314e9f179","Type":"ContainerDied","Data":"329d8138d4635f4a559e3331f5633e9c9d3d847e6c40de9596546311c934be6f"} Sep 07 03:48:48.309717 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.309684 2555 scope.go:117] "RemoveContainer" containerID="e13ecd3ad070993f611e095b78d47e51f0cc24c00560acaa2b969adbc1cae176" Sep 07 03:48:48.309942 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.309926 2555 scope.go:117] "RemoveContainer" containerID="329d8138d4635f4a559e3331f5633e9c9d3d847e6c40de9596546311c934be6f" Sep 07 03:48:48.310141 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:48.310118 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zvrg9_openshift-console-operator(577e74b9-769c-48fa-b70f-e87314e9f179)\"" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" podUID="577e74b9-769c-48fa-b70f-e87314e9f179" Sep 07 03:48:48.374933 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.374915 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t" Sep 07 03:48:48.387833 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.387818 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6wvfr_b83fecd8-da21-48bf-a8f4-13c33d46742a/dns-node-resolver/0.log" Sep 07 03:48:48.392635 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.392612 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:48.446641 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.446505 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:48.517283 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.517216 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-2j9wk"] Sep 07 03:48:48.550867 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.550841 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:48:48.550867 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.550854 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-2j9wk"] Sep 07 03:48:48.553288 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.553247 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 07 03:48:48.553288 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.553282 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 07 03:48:48.553453 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.553323 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9zjqr\"" Sep 07 03:48:48.622461 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.622433 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:48.637317 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.637294 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:48:48.637377 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.637325 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-nginx-conf\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:48:48.676615 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.676597 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t"] Sep 07 03:48:48.679742 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.679719 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:48.679926 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:48.679905 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10582104_f53f_40dd_9f07_81d404420cd4.slice/crio-27b5bc4d0bbbe78148f10e7c2a5cb98456e076a8b42aaeb191289bdd37e4f91b WatchSource:0}: Error finding container 27b5bc4d0bbbe78148f10e7c2a5cb98456e076a8b42aaeb191289bdd37e4f91b: Status 404 returned error can't find the container with id 27b5bc4d0bbbe78148f10e7c2a5cb98456e076a8b42aaeb191289bdd37e4f91b Sep 07 03:48:48.738570 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.738549 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:48:48.738662 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.738584 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-nginx-conf\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:48:48.738710 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:48.738678 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 03:48:48.738746 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:48.738731 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert podName:3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:49.238713783 +0000 UTC m=+47.813580050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2j9wk" (UID: "3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6") : secret "networking-console-plugin-cert" not found Sep 07 03:48:48.739138 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.739123 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-nginx-conf\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:48:48.740945 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.740915 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:48.844077 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.844009 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:48.902363 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:48.902337 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:49.141965 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.141899 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:49.141965 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.141941 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:49.142342 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:49.142113 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 03:48:49.142342 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:49.142137 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 03:48:49.142342 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:49.142179 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls podName:dd7656b6-1017-4966-92af-fd1d32d08a7f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:51.142160901 +0000 UTC m=+49.717027192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wv9z9" (UID: "dd7656b6-1017-4966-92af-fd1d32d08a7f") : secret "insights-runtime-extractor-tls" not found Sep 07 03:48:49.142342 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:49.142220 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm podName:dd7656b6-1017-4966-92af-fd1d32d08a7f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:51.142202944 +0000 UTC m=+49.717069233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wv9z9" (UID: "dd7656b6-1017-4966-92af-fd1d32d08a7f") : configmap "kube-rbac-proxy" not found Sep 07 03:48:49.242545 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.242500 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:48:49.242699 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:49.242647 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 03:48:49.242763 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:49.242715 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert podName:3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:50.242696022 +0000 UTC m=+48.817562290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2j9wk" (UID: "3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6") : secret "networking-console-plugin-cert" not found Sep 07 03:48:49.314695 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.314668 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 03:48:49.315128 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.315085 2555 scope.go:117] "RemoveContainer" containerID="329d8138d4635f4a559e3331f5633e9c9d3d847e6c40de9596546311c934be6f" Sep 07 03:48:49.315275 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:49.315246 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zvrg9_openshift-console-operator(577e74b9-769c-48fa-b70f-e87314e9f179)\"" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" podUID="577e74b9-769c-48fa-b70f-e87314e9f179" Sep 07 03:48:49.316234 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.316204 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t" event={"ID":"10582104-f53f-40dd-9f07-81d404420cd4","Type":"ContainerStarted","Data":"27b5bc4d0bbbe78148f10e7c2a5cb98456e076a8b42aaeb191289bdd37e4f91b"} Sep 07 03:48:49.391475 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.391447 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fst8d_ca804fd3-f751-4640-8e5a-a831ab9e2f55/node-ca/0.log" Sep 07 03:48:49.399519 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.399447 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-cx8x8"] Sep 07 03:48:49.420143 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.420123 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-cx8x8"] Sep 07 03:48:49.420252 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.420202 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:49.422715 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.422628 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 07 03:48:49.422715 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.422628 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 07 03:48:49.422859 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.422740 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 07 03:48:49.422900 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.422859 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-z9vsg\"" Sep 07 03:48:49.422900 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.422874 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 07 03:48:49.435811 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.435787 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:49.493085 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.493046 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:49.545490 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.545469 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d14aadd7-2337-4eb7-bc58-310f027918b2-signing-cabundle\") pod \"service-ca-75d99c484-cx8x8\" (UID: \"d14aadd7-2337-4eb7-bc58-310f027918b2\") " pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:49.545648 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.545621 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznfx\" (UniqueName: \"kubernetes.io/projected/d14aadd7-2337-4eb7-bc58-310f027918b2-kube-api-access-vznfx\") pod \"service-ca-75d99c484-cx8x8\" (UID: \"d14aadd7-2337-4eb7-bc58-310f027918b2\") " pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:49.545761 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.545669 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d14aadd7-2337-4eb7-bc58-310f027918b2-signing-key\") pod \"service-ca-75d99c484-cx8x8\" (UID: \"d14aadd7-2337-4eb7-bc58-310f027918b2\") " pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:49.647135 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.647104 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznfx\" (UniqueName: \"kubernetes.io/projected/d14aadd7-2337-4eb7-bc58-310f027918b2-kube-api-access-vznfx\") pod \"service-ca-75d99c484-cx8x8\" (UID: \"d14aadd7-2337-4eb7-bc58-310f027918b2\") " pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:49.647285 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.647154 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d14aadd7-2337-4eb7-bc58-310f027918b2-signing-key\") pod \"service-ca-75d99c484-cx8x8\" (UID: \"d14aadd7-2337-4eb7-bc58-310f027918b2\") " pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:49.647463 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.647439 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d14aadd7-2337-4eb7-bc58-310f027918b2-signing-cabundle\") pod \"service-ca-75d99c484-cx8x8\" (UID: \"d14aadd7-2337-4eb7-bc58-310f027918b2\") " pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:49.648112 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.648093 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d14aadd7-2337-4eb7-bc58-310f027918b2-signing-cabundle\") pod \"service-ca-75d99c484-cx8x8\" (UID: \"d14aadd7-2337-4eb7-bc58-310f027918b2\") " pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:49.649767 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.649714 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d14aadd7-2337-4eb7-bc58-310f027918b2-signing-key\") pod \"service-ca-75d99c484-cx8x8\" (UID: \"d14aadd7-2337-4eb7-bc58-310f027918b2\") " pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:49.656547 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.656526 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznfx\" (UniqueName: \"kubernetes.io/projected/d14aadd7-2337-4eb7-bc58-310f027918b2-kube-api-access-vznfx\") pod \"service-ca-75d99c484-cx8x8\" (UID: \"d14aadd7-2337-4eb7-bc58-310f027918b2\") " pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:49.729215 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:49.729175 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-cx8x8" Sep 07 03:48:50.048718 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:50.048693 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-cx8x8"] Sep 07 03:48:50.152898 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:48:50.152863 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14aadd7_2337_4eb7_bc58_310f027918b2.slice/crio-346ce051fdcdb9a20517eb62ed5da2de59b5ed389995af9f1ef688d540c0dde8 WatchSource:0}: Error finding container 346ce051fdcdb9a20517eb62ed5da2de59b5ed389995af9f1ef688d540c0dde8: Status 404 returned error can't find the container with id 346ce051fdcdb9a20517eb62ed5da2de59b5ed389995af9f1ef688d540c0dde8 Sep 07 03:48:50.189911 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:50.189894 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gljg7_212b6720-24d5-46df-8053-6859f2e790ca/serve-healthcheck-canary/0.log" Sep 07 03:48:50.252021 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:50.251997 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:48:50.252166 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:50.252147 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 03:48:50.252217 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:50.252204 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert podName:3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:52.252190334 +0000 UTC m=+50.827056616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2j9wk" (UID: "3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6") : secret "networking-console-plugin-cert" not found Sep 07 03:48:50.321719 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:50.321690 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t" event={"ID":"10582104-f53f-40dd-9f07-81d404420cd4","Type":"ContainerStarted","Data":"a5a7b0aa7e60364d549f03a7cc325e4db20a6b39d82ea177a2ac22d628e81511"} Sep 07 03:48:50.323529 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:50.323506 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-cx8x8" event={"ID":"d14aadd7-2337-4eb7-bc58-310f027918b2","Type":"ContainerStarted","Data":"52fea1d07568c8f4e8e199315c7fd9a34ed7b8f980ba4eff625d405153e45b5d"} Sep 07 03:48:50.323639 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:50.323551 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-cx8x8" event={"ID":"d14aadd7-2337-4eb7-bc58-310f027918b2","Type":"ContainerStarted","Data":"346ce051fdcdb9a20517eb62ed5da2de59b5ed389995af9f1ef688d540c0dde8"} Sep 07 03:48:50.534814 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:50.534779 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:51.159005 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:51.158969 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:51.159190 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:51.159004 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:51.159190 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:51.159110 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 03:48:51.159190 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:51.159165 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 03:48:51.159190 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:51.159179 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm podName:dd7656b6-1017-4966-92af-fd1d32d08a7f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:55.159163332 +0000 UTC m=+53.734029600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wv9z9" (UID: "dd7656b6-1017-4966-92af-fd1d32d08a7f") : configmap "kube-rbac-proxy" not found Sep 07 03:48:51.159420 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:51.159211 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls podName:dd7656b6-1017-4966-92af-fd1d32d08a7f nodeName:}" failed. No retries permitted until 2026-09-07 03:48:55.1591996 +0000 UTC m=+53.734065882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wv9z9" (UID: "dd7656b6-1017-4966-92af-fd1d32d08a7f") : secret "insights-runtime-extractor-tls" not found Sep 07 03:48:51.328144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:51.328109 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t" event={"ID":"10582104-f53f-40dd-9f07-81d404420cd4","Type":"ContainerStarted","Data":"11cf1af9a6ba0e4391a39ff358752c307c28ea86527983eed52cad1b2ce84eb6"} Sep 07 03:48:51.346284 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:51.346227 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-vnd5t" podStartSLOduration=1.843843192 podStartE2EDuration="3.346209993s" totalImagesPullingTime="1.502366801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:51.34581868 +0000 UTC m=+49.920684971" watchObservedRunningTime="2026-09-07 03:48:51.346209993 +0000 UTC m=+49.921076281" Sep 07 03:48:51.362026 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:51.361983 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-cx8x8" podStartSLOduration=2.361968886 podStartE2EDuration="2.361968886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:51.361232778 +0000 UTC m=+49.936099080" watchObservedRunningTime="2026-09-07 03:48:51.361968886 +0000 UTC m=+49.936835174" Sep 07 03:48:52.072457 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.072422 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:52.239773 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.239738 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:52.268102 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.268045 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:48:52.268251 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.268203 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 03:48:52.268307 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.268261 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert podName:3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:48:56.268243415 +0000 UTC m=+54.843109696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2j9wk" (UID: "3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6") : secret "networking-console-plugin-cert" not found Sep 07 03:48:52.387119 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.387030 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:52.458949 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.458917 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:52.515814 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.515780 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:52.616275 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.616230 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:52.671045 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.670960 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:48:52.671045 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.671008 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:48:52.671045 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.671037 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:48:52.671293 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.671111 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:52.671293 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.671142 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:52.671293 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.671156 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 03:48:52.671293 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.671173 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7f8788ff-2zcb2: secret "image-registry-tls" not found Sep 07 03:48:52.671293 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.671213 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:48:52.671293 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.671230 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls podName:66017c83-f8c1-4280-aefa-499d9861a68f nodeName:}" failed. No retries permitted until 2026-09-07 03:49:08.671210644 +0000 UTC m=+67.246076934 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls") pod "image-registry-7b7f8788ff-2zcb2" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f") : secret "image-registry-tls" not found Sep 07 03:48:52.671293 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.671156 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:52.671293 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.671275 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 03:48:52.671293 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.671160 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 03:48:52.671621 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.671302 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls podName:8b8f75fd-b50c-461a-9f2f-d8433861d39a nodeName:}" failed. No retries permitted until 2026-09-07 03:49:08.671285673 +0000 UTC m=+67.246151954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5qh6c" (UID: "8b8f75fd-b50c-461a-9f2f-d8433861d39a") : secret "cluster-monitoring-operator-tls" not found Sep 07 03:48:52.671621 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.671323 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:49:08.671311581 +0000 UTC m=+67.246177873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : secret "router-metrics-certs-default" not found Sep 07 03:48:52.671621 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.671348 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle podName:cc36ab08-3dc2-4011-b24a-4ee2635c4630 nodeName:}" failed. No retries permitted until 2026-09-07 03:49:08.671330378 +0000 UTC m=+67.246196656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle") pod "router-default-7447dd5b74-p7pm5" (UID: "cc36ab08-3dc2-4011-b24a-4ee2635c4630") : configmap references non-existent config key: service-ca.crt Sep 07 03:48:52.671621 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:52.671410 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls podName:08336dd4-dd4c-47e3-ba8d-531ca667a0b5 nodeName:}" failed. No retries permitted until 2026-09-07 03:49:08.671398592 +0000 UTC m=+67.246264865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls") pod "dns-default-rh982" (UID: "08336dd4-dd4c-47e3-ba8d-531ca667a0b5") : secret "dns-default-metrics-tls" not found Sep 07 03:48:52.673968 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.673935 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1503ab01-93bb-4c30-95ec-e8c7fb37b620-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2n4bk\" (UID: \"1503ab01-93bb-4c30-95ec-e8c7fb37b620\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:52.677734 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.677711 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" Sep 07 03:48:52.912149 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.912111 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:52.980841 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.980815 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk"] Sep 07 03:48:52.983310 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:52.983282 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:53.181304 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:53.181270 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:53.338217 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:53.338151 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" event={"ID":"1503ab01-93bb-4c30-95ec-e8c7fb37b620","Type":"ContainerStarted","Data":"6332c59c7ed203e298f414f415649fe810290195f0a62b7fb21332f14eaeb78b"} Sep 07 03:48:53.451197 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:53.451163 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:54.169455 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:54.169424 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:54.568481 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:54.568421 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:54.765572 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:54.765537 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:54.964136 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:54.964090 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:55.197963 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:55.197884 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:55.197963 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:55.197927 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:55.198200 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:55.198123 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 03:48:55.198200 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:55.198197 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls podName:dd7656b6-1017-4966-92af-fd1d32d08a7f nodeName:}" failed. No retries permitted until 2026-09-07 03:49:03.198175435 +0000 UTC m=+61.773041719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wv9z9" (UID: "dd7656b6-1017-4966-92af-fd1d32d08a7f") : secret "insights-runtime-extractor-tls" not found Sep 07 03:48:55.198432 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:55.198415 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd7656b6-1017-4966-92af-fd1d32d08a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:48:55.235539 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:55.235505 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" limit=1024 Sep 07 03:48:55.346836 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:55.346805 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" event={"ID":"1503ab01-93bb-4c30-95ec-e8c7fb37b620","Type":"ContainerStarted","Data":"7f5a46aa50c33022baf6182beb2e96fe4692c25fb9e348093ec3619dd223afc8"} Sep 07 03:48:55.346979 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:55.346904 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" event={"ID":"1503ab01-93bb-4c30-95ec-e8c7fb37b620","Type":"ContainerStarted","Data":"7f2f94c17fe98a8a0e6b58df962411b4894e3e21dd0adb87a2d8c74c4c0c8d22"} Sep 07 03:48:55.378122 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:55.378082 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2n4bk" podStartSLOduration=38.467787653 podStartE2EDuration="40.37806794s" totalImagesPullingTime="1.910280287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:48:55.377028961 +0000 UTC m=+53.951895252" watchObservedRunningTime="2026-09-07 03:48:55.37806794 +0000 UTC m=+53.952934223" Sep 07 03:48:56.306317 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:56.306279 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:48:56.306738 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:56.306435 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 03:48:56.306738 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:56.306507 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert podName:3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6 nodeName:}" failed. No retries permitted until 2026-09-07 03:49:04.306489909 +0000 UTC m=+62.881356200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2j9wk" (UID: "3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6") : secret "networking-console-plugin-cert" not found Sep 07 03:48:57.041643 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:57.041606 2555 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:57.041826 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:57.041660 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:48:57.041988 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:48:57.041975 2555 scope.go:117] "RemoveContainer" containerID="329d8138d4635f4a559e3331f5633e9c9d3d847e6c40de9596546311c934be6f" Sep 07 03:48:57.042161 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:48:57.042142 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zvrg9_openshift-console-operator(577e74b9-769c-48fa-b70f-e87314e9f179)\"" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" podUID="577e74b9-769c-48fa-b70f-e87314e9f179" Sep 07 03:49:02.181321 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:02.181282 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xfkc8" Sep 07 03:49:03.268848 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:03.268810 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:49:03.271050 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:03.271030 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd7656b6-1017-4966-92af-fd1d32d08a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wv9z9\" (UID: \"dd7656b6-1017-4966-92af-fd1d32d08a7f\") " pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:49:03.336934 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:03.336905 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7k4qm\"" Sep 07 03:49:03.344926 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:03.344896 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wv9z9" Sep 07 03:49:03.617810 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:03.617732 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wv9z9"] Sep 07 03:49:03.620886 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:03.620861 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7656b6_1017_4966_92af_fd1d32d08a7f.slice/crio-b29b50345e071acbd29a7c8330d8a05ec600d7b06a2e2f8a2a86c51bffa24cc4 WatchSource:0}: Error finding container b29b50345e071acbd29a7c8330d8a05ec600d7b06a2e2f8a2a86c51bffa24cc4: Status 404 returned error can't find the container with id b29b50345e071acbd29a7c8330d8a05ec600d7b06a2e2f8a2a86c51bffa24cc4 Sep 07 03:49:04.378274 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:04.378243 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wv9z9" event={"ID":"dd7656b6-1017-4966-92af-fd1d32d08a7f","Type":"ContainerStarted","Data":"d5752155e6f5c699936af5871d78e8a861325d3c9f44633713a5390b26221a8d"} Sep 07 03:49:04.378548 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:04.378304 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wv9z9" event={"ID":"dd7656b6-1017-4966-92af-fd1d32d08a7f","Type":"ContainerStarted","Data":"64432d280155c9d9d5912a5803ba4f1fdc5a411b486b9cbbfba36c5af8d418d2"} Sep 07 03:49:04.378548 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:04.378323 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wv9z9" event={"ID":"dd7656b6-1017-4966-92af-fd1d32d08a7f","Type":"ContainerStarted","Data":"b29b50345e071acbd29a7c8330d8a05ec600d7b06a2e2f8a2a86c51bffa24cc4"} Sep 07 03:49:04.378548 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:04.378379 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:49:04.380434 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:04.380412 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2j9wk\" (UID: \"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:49:04.462187 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:04.462158 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9zjqr\"" Sep 07 03:49:04.469977 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:04.469956 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" Sep 07 03:49:04.736213 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:04.736050 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-2j9wk"] Sep 07 03:49:04.738518 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:04.738487 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9cf8b6_a25a_4e43_9299_aba1b9ddf7d6.slice/crio-64af6633303528c042de98be1bcf1fe6c116e433e695eb5d1ec5ffe63685224a WatchSource:0}: Error finding container 64af6633303528c042de98be1bcf1fe6c116e433e695eb5d1ec5ffe63685224a: Status 404 returned error can't find the container with id 64af6633303528c042de98be1bcf1fe6c116e433e695eb5d1ec5ffe63685224a Sep 07 03:49:05.386228 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:05.386177 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" event={"ID":"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6","Type":"ContainerStarted","Data":"64af6633303528c042de98be1bcf1fe6c116e433e695eb5d1ec5ffe63685224a"} Sep 07 03:49:06.390620 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:06.390538 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" event={"ID":"3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6","Type":"ContainerStarted","Data":"dd7f241d76277c35a8b6c02f7d3293b86e9fa25d20629501d929c3d69716095f"} Sep 07 03:49:06.392691 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:06.392667 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wv9z9" event={"ID":"dd7656b6-1017-4966-92af-fd1d32d08a7f","Type":"ContainerStarted","Data":"49d1f510d421861f923bc0e122398b500b7ca98e4511104b971554e708446c5d"} Sep 07 03:49:06.407613 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:06.407568 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-2j9wk" podStartSLOduration=17.07546065 podStartE2EDuration="18.407552603s" totalImagesPullingTime="1.332091953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:06.406632419 +0000 UTC m=+64.981498707" watchObservedRunningTime="2026-09-07 03:49:06.407552603 +0000 UTC m=+64.982418892" Sep 07 03:49:06.441543 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:06.441498 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wv9z9" podStartSLOduration=17.103738223 podStartE2EDuration="19.441484673s" totalImagesPullingTime="2.33774645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:06.441006051 +0000 UTC m=+65.015872340" watchObservedRunningTime="2026-09-07 03:49:06.441484673 +0000 UTC m=+65.016350963" Sep 07 03:49:07.607991 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.607954 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:49:07.610773 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.610755 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 03:49:07.620901 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.620872 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a784083-f7eb-4508-9ef5-07ae8c51a973-original-pull-secret\") pod \"global-pull-secret-syncer-gbx7k\" (UID: \"4a784083-f7eb-4508-9ef5-07ae8c51a973\") " pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:49:07.708861 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.708830 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk4m\" (UniqueName: \"kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m\") pod \"network-check-target-x6689\" (UID: \"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6\") " pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:49:07.709024 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.708927 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:49:07.711374 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.711344 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zk4m\" (UniqueName: \"kubernetes.io/projected/1c70ff1e-fc11-4f58-80b5-98a9901fd7d6-kube-api-access-7zk4m\") pod \"network-check-target-x6689\" (UID: \"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6\") " pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:49:07.711473 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.711354 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 03:49:07.721708 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.721687 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d176a6b-c22a-4f6a-9551-1dbd25928cf8-metrics-certs\") pod \"network-metrics-daemon-sxp6n\" (UID: \"9d176a6b-c22a-4f6a-9551-1dbd25928cf8\") " pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:49:07.841424 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.841398 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbx7k" Sep 07 03:49:07.965422 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.965398 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wrs9m\"" Sep 07 03:49:07.972339 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.972314 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-25ldn\"" Sep 07 03:49:07.973352 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.973337 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:49:07.981102 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:07.981084 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxp6n" Sep 07 03:49:08.133760 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.133682 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gbx7k"] Sep 07 03:49:08.136457 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:08.136430 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a784083_f7eb_4508_9ef5_07ae8c51a973.slice/crio-6d01cf9b716c0c73adc4b1c576033b829b002ce3a04edc305c25cb471dd90c53 WatchSource:0}: Error finding container 6d01cf9b716c0c73adc4b1c576033b829b002ce3a04edc305c25cb471dd90c53: Status 404 returned error can't find the container with id 6d01cf9b716c0c73adc4b1c576033b829b002ce3a04edc305c25cb471dd90c53 Sep 07 03:49:08.261857 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.261829 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x6689"] Sep 07 03:49:08.264421 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:08.264400 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c70ff1e_fc11_4f58_80b5_98a9901fd7d6.slice/crio-72dd197717f88da8df94d8b501eeb52f3b57dc6d6ec27e61b6cd9aa97aa0857b WatchSource:0}: Error finding container 72dd197717f88da8df94d8b501eeb52f3b57dc6d6ec27e61b6cd9aa97aa0857b: Status 404 returned error can't find the container with id 72dd197717f88da8df94d8b501eeb52f3b57dc6d6ec27e61b6cd9aa97aa0857b Sep 07 03:49:08.289998 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.289971 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sxp6n"] Sep 07 03:49:08.293967 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:08.293942 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d176a6b_c22a_4f6a_9551_1dbd25928cf8.slice/crio-43cd860ad0bab7907f8e5dad51cb0369ed0e4859e0f3108081880e34d4836093 WatchSource:0}: Error finding container 43cd860ad0bab7907f8e5dad51cb0369ed0e4859e0f3108081880e34d4836093: Status 404 returned error can't find the container with id 43cd860ad0bab7907f8e5dad51cb0369ed0e4859e0f3108081880e34d4836093 Sep 07 03:49:08.400553 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.400479 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x6689" event={"ID":"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6","Type":"ContainerStarted","Data":"b038a78b52658f0601ccb2ff9e86df610ba67fedc08bde874f8929cb8ef2ca81"} Sep 07 03:49:08.400694 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.400635 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:49:08.400694 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.400676 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x6689" event={"ID":"1c70ff1e-fc11-4f58-80b5-98a9901fd7d6","Type":"ContainerStarted","Data":"72dd197717f88da8df94d8b501eeb52f3b57dc6d6ec27e61b6cd9aa97aa0857b"} Sep 07 03:49:08.401654 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.401634 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gbx7k" event={"ID":"4a784083-f7eb-4508-9ef5-07ae8c51a973","Type":"ContainerStarted","Data":"6d01cf9b716c0c73adc4b1c576033b829b002ce3a04edc305c25cb471dd90c53"} Sep 07 03:49:08.402714 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.402684 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxp6n" event={"ID":"9d176a6b-c22a-4f6a-9551-1dbd25928cf8","Type":"ContainerStarted","Data":"43cd860ad0bab7907f8e5dad51cb0369ed0e4859e0f3108081880e34d4836093"} Sep 07 03:49:08.418672 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.418630 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x6689" podStartSLOduration=66.418616724 podStartE2EDuration="1m6.418616724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:08.417496658 +0000 UTC m=+66.992362960" watchObservedRunningTime="2026-09-07 03:49:08.418616724 +0000 UTC m=+66.993483013" Sep 07 03:49:08.718764 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.718730 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:49:08.719192 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.718779 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:49:08.719192 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.718828 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:49:08.719192 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.718848 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:49:08.719192 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.718876 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:49:08.719542 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.719517 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc36ab08-3dc2-4011-b24a-4ee2635c4630-service-ca-bundle\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:49:08.721754 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.721731 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08336dd4-dd4c-47e3-ba8d-531ca667a0b5-metrics-tls\") pod \"dns-default-rh982\" (UID: \"08336dd4-dd4c-47e3-ba8d-531ca667a0b5\") " pod="openshift-dns/dns-default-rh982" Sep 07 03:49:08.721865 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.721785 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc36ab08-3dc2-4011-b24a-4ee2635c4630-metrics-certs\") pod \"router-default-7447dd5b74-p7pm5\" (UID: \"cc36ab08-3dc2-4011-b24a-4ee2635c4630\") " pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:49:08.721865 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.721809 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") pod \"image-registry-7b7f8788ff-2zcb2\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:49:08.721942 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.721859 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8f75fd-b50c-461a-9f2f-d8433861d39a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5qh6c\" (UID: \"8b8f75fd-b50c-461a-9f2f-d8433861d39a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:49:08.852167 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.851929 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dgwxt\"" Sep 07 03:49:08.860168 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.860133 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:49:08.868629 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.868607 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zwwvl\"" Sep 07 03:49:08.876704 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.876667 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" Sep 07 03:49:08.886180 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.886160 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-njlgh\"" Sep 07 03:49:08.900103 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.894449 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:49:08.919835 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.919604 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n9q8q\"" Sep 07 03:49:08.927290 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:08.927262 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rh982" Sep 07 03:49:09.172730 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.172702 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c"] Sep 07 03:49:09.176109 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.176043 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7447dd5b74-p7pm5"] Sep 07 03:49:09.210910 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.210843 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b7f8788ff-2zcb2"] Sep 07 03:49:09.255886 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.255806 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rh982"] Sep 07 03:49:09.431663 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:09.431627 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc36ab08_3dc2_4011_b24a_4ee2635c4630.slice/crio-64a22fe05e11ab60c0c7c3e35f9a828438e04aacffbb4ff720ae38b50c751554 WatchSource:0}: Error finding container 64a22fe05e11ab60c0c7c3e35f9a828438e04aacffbb4ff720ae38b50c751554: Status 404 returned error can't find the container with id 64a22fe05e11ab60c0c7c3e35f9a828438e04aacffbb4ff720ae38b50c751554 Sep 07 03:49:09.432596 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:09.432440 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08336dd4_dd4c_47e3_ba8d_531ca667a0b5.slice/crio-47371d770bca67038b06bc941aacca5b01a2bbfd021e32db6e59814f0aa1565a WatchSource:0}: Error finding container 47371d770bca67038b06bc941aacca5b01a2bbfd021e32db6e59814f0aa1565a: Status 404 returned error can't find the container with id 47371d770bca67038b06bc941aacca5b01a2bbfd021e32db6e59814f0aa1565a Sep 07 03:49:09.433354 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:09.433301 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8f75fd_b50c_461a_9f2f_d8433861d39a.slice/crio-66e51cfda78422776a780d9d227f7afc1e7e96ec61cb448fce656c086e70a7fb WatchSource:0}: Error finding container 66e51cfda78422776a780d9d227f7afc1e7e96ec61cb448fce656c086e70a7fb: Status 404 returned error can't find the container with id 66e51cfda78422776a780d9d227f7afc1e7e96ec61cb448fce656c086e70a7fb Sep 07 03:49:09.434539 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:09.434504 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66017c83_f8c1_4280_aefa_499d9861a68f.slice/crio-4684ef973366e7f25dff7f16287bff139b876d5906dc01b24eab686f78589d82 WatchSource:0}: Error finding container 4684ef973366e7f25dff7f16287bff139b876d5906dc01b24eab686f78589d82: Status 404 returned error can't find the container with id 4684ef973366e7f25dff7f16287bff139b876d5906dc01b24eab686f78589d82 Sep 07 03:49:09.510354 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.510328 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj"] Sep 07 03:49:09.516182 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.516157 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq"] Sep 07 03:49:09.516328 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.516304 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.520136 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.519780 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 07 03:49:09.520136 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.520005 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 07 03:49:09.521131 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.520410 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 07 03:49:09.521131 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.520535 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 07 03:49:09.521131 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.520727 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 07 03:49:09.521131 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.520933 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 07 03:49:09.521389 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.521161 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76"] Sep 07 03:49:09.522628 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.521489 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 07 03:49:09.522628 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.521626 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.533536 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.524890 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" Sep 07 03:49:09.533536 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.527223 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 07 03:49:09.533536 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.530106 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7hgx5\"" Sep 07 03:49:09.533536 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.530341 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 07 03:49:09.533536 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.530573 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq"] Sep 07 03:49:09.533536 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.530611 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76"] Sep 07 03:49:09.533536 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.531495 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj"] Sep 07 03:49:09.631870 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.631797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.631870 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.631837 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-ca\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.632050 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.631870 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.632050 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.631907 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a15770ff-4db9-4dc4-8971-a7eeb7953137-tmp\") pod \"klusterlet-addon-workmgr-d67ccdcb9-nhgsq\" (UID: \"a15770ff-4db9-4dc4-8971-a7eeb7953137\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.632050 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.631933 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xscxs\" (UniqueName: \"kubernetes.io/projected/6c4c4546-b1dd-431c-a4d1-0144d4b698d7-kube-api-access-xscxs\") pod \"managed-serviceaccount-addon-agent-78889d8498-z5w76\" (UID: \"6c4c4546-b1dd-431c-a4d1-0144d4b698d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" Sep 07 03:49:09.632050 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.631963 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29hks\" (UniqueName: \"kubernetes.io/projected/a15770ff-4db9-4dc4-8971-a7eeb7953137-kube-api-access-29hks\") pod \"klusterlet-addon-workmgr-d67ccdcb9-nhgsq\" (UID: \"a15770ff-4db9-4dc4-8971-a7eeb7953137\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.632050 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.631990 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6c4c4546-b1dd-431c-a4d1-0144d4b698d7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78889d8498-z5w76\" (UID: \"6c4c4546-b1dd-431c-a4d1-0144d4b698d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" Sep 07 03:49:09.632050 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.632017 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a15770ff-4db9-4dc4-8971-a7eeb7953137-klusterlet-config\") pod \"klusterlet-addon-workmgr-d67ccdcb9-nhgsq\" (UID: \"a15770ff-4db9-4dc4-8971-a7eeb7953137\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.632337 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.632072 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26005b5-a488-4b37-bb20-a11934c65b28-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.632337 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.632097 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxm5\" (UniqueName: \"kubernetes.io/projected/d26005b5-a488-4b37-bb20-a11934c65b28-kube-api-access-fwxm5\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.632337 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.632124 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-hub\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.732894 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.732944 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a15770ff-4db9-4dc4-8971-a7eeb7953137-tmp\") pod \"klusterlet-addon-workmgr-d67ccdcb9-nhgsq\" (UID: \"a15770ff-4db9-4dc4-8971-a7eeb7953137\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.732973 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xscxs\" (UniqueName: \"kubernetes.io/projected/6c4c4546-b1dd-431c-a4d1-0144d4b698d7-kube-api-access-xscxs\") pod \"managed-serviceaccount-addon-agent-78889d8498-z5w76\" (UID: \"6c4c4546-b1dd-431c-a4d1-0144d4b698d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.733003 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29hks\" (UniqueName: \"kubernetes.io/projected/a15770ff-4db9-4dc4-8971-a7eeb7953137-kube-api-access-29hks\") pod \"klusterlet-addon-workmgr-d67ccdcb9-nhgsq\" (UID: \"a15770ff-4db9-4dc4-8971-a7eeb7953137\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.733029 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6c4c4546-b1dd-431c-a4d1-0144d4b698d7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78889d8498-z5w76\" (UID: \"6c4c4546-b1dd-431c-a4d1-0144d4b698d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.733051 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a15770ff-4db9-4dc4-8971-a7eeb7953137-klusterlet-config\") pod \"klusterlet-addon-workmgr-d67ccdcb9-nhgsq\" (UID: \"a15770ff-4db9-4dc4-8971-a7eeb7953137\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.733117 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26005b5-a488-4b37-bb20-a11934c65b28-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.733140 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxm5\" (UniqueName: \"kubernetes.io/projected/d26005b5-a488-4b37-bb20-a11934c65b28-kube-api-access-fwxm5\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.733163 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-hub\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.733251 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.733275 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-ca\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.734354 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26005b5-a488-4b37-bb20-a11934c65b28-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.734931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.734515 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a15770ff-4db9-4dc4-8971-a7eeb7953137-tmp\") pod \"klusterlet-addon-workmgr-d67ccdcb9-nhgsq\" (UID: \"a15770ff-4db9-4dc4-8971-a7eeb7953137\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.738144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.738099 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.739458 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.738593 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.739458 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.738882 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6c4c4546-b1dd-431c-a4d1-0144d4b698d7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78889d8498-z5w76\" (UID: \"6c4c4546-b1dd-431c-a4d1-0144d4b698d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" Sep 07 03:49:09.739458 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.739335 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-ca\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.739458 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.739419 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a15770ff-4db9-4dc4-8971-a7eeb7953137-klusterlet-config\") pod \"klusterlet-addon-workmgr-d67ccdcb9-nhgsq\" (UID: \"a15770ff-4db9-4dc4-8971-a7eeb7953137\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.739844 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.739808 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d26005b5-a488-4b37-bb20-a11934c65b28-hub\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.748954 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.748930 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29hks\" (UniqueName: \"kubernetes.io/projected/a15770ff-4db9-4dc4-8971-a7eeb7953137-kube-api-access-29hks\") pod \"klusterlet-addon-workmgr-d67ccdcb9-nhgsq\" (UID: \"a15770ff-4db9-4dc4-8971-a7eeb7953137\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.761401 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.754924 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xscxs\" (UniqueName: \"kubernetes.io/projected/6c4c4546-b1dd-431c-a4d1-0144d4b698d7-kube-api-access-xscxs\") pod \"managed-serviceaccount-addon-agent-78889d8498-z5w76\" (UID: \"6c4c4546-b1dd-431c-a4d1-0144d4b698d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" Sep 07 03:49:09.761401 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.755512 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxm5\" (UniqueName: \"kubernetes.io/projected/d26005b5-a488-4b37-bb20-a11934c65b28-kube-api-access-fwxm5\") pod \"cluster-proxy-proxy-agent-84d98fc695-pqcqj\" (UID: \"d26005b5-a488-4b37-bb20-a11934c65b28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.842860 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.842821 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" Sep 07 03:49:09.853679 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.853493 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:09.861961 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:09.861933 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" Sep 07 03:49:10.138082 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.138016 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj"] Sep 07 03:49:10.141196 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:10.141134 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd26005b5_a488_4b37_bb20_a11934c65b28.slice/crio-e868bd15c402dffb23b43ee48c604ff8c37011dd0bb28d2ea67f0b0a40a8d3d4 WatchSource:0}: Error finding container e868bd15c402dffb23b43ee48c604ff8c37011dd0bb28d2ea67f0b0a40a8d3d4: Status 404 returned error can't find the container with id e868bd15c402dffb23b43ee48c604ff8c37011dd0bb28d2ea67f0b0a40a8d3d4 Sep 07 03:49:10.149888 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.149860 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq"] Sep 07 03:49:10.153437 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:10.153413 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15770ff_4db9_4dc4_8971_a7eeb7953137.slice/crio-ce4c748f842c6be547dc01da83afda5e41a5280a45839b2059d8404aa9cfc75c WatchSource:0}: Error finding container ce4c748f842c6be547dc01da83afda5e41a5280a45839b2059d8404aa9cfc75c: Status 404 returned error can't find the container with id ce4c748f842c6be547dc01da83afda5e41a5280a45839b2059d8404aa9cfc75c Sep 07 03:49:10.167147 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.167117 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76"] Sep 07 03:49:10.171020 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:10.170992 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c4c4546_b1dd_431c_a4d1_0144d4b698d7.slice/crio-23abe8303778f5789e39d5613d65bb999263aaaa7388f4330493a7d84c0ea64d WatchSource:0}: Error finding container 23abe8303778f5789e39d5613d65bb999263aaaa7388f4330493a7d84c0ea64d: Status 404 returned error can't find the container with id 23abe8303778f5789e39d5613d65bb999263aaaa7388f4330493a7d84c0ea64d Sep 07 03:49:10.415047 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.414963 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" event={"ID":"a15770ff-4db9-4dc4-8971-a7eeb7953137","Type":"ContainerStarted","Data":"ce4c748f842c6be547dc01da83afda5e41a5280a45839b2059d8404aa9cfc75c"} Sep 07 03:49:10.416803 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.416763 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" event={"ID":"6c4c4546-b1dd-431c-a4d1-0144d4b698d7","Type":"ContainerStarted","Data":"23abe8303778f5789e39d5613d65bb999263aaaa7388f4330493a7d84c0ea64d"} Sep 07 03:49:10.418340 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.418294 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" event={"ID":"d26005b5-a488-4b37-bb20-a11934c65b28","Type":"ContainerStarted","Data":"e868bd15c402dffb23b43ee48c604ff8c37011dd0bb28d2ea67f0b0a40a8d3d4"} Sep 07 03:49:10.420307 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.420233 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" event={"ID":"66017c83-f8c1-4280-aefa-499d9861a68f","Type":"ContainerStarted","Data":"3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb"} Sep 07 03:49:10.420307 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.420277 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" event={"ID":"66017c83-f8c1-4280-aefa-499d9861a68f","Type":"ContainerStarted","Data":"4684ef973366e7f25dff7f16287bff139b876d5906dc01b24eab686f78589d82"} Sep 07 03:49:10.420777 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.420509 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:49:10.425855 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.425329 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxp6n" event={"ID":"9d176a6b-c22a-4f6a-9551-1dbd25928cf8","Type":"ContainerStarted","Data":"daf5276bf672970016e9f2ccf8ef3b08ce08cd6072c664f0be072bde55c34d9e"} Sep 07 03:49:10.425855 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.425378 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxp6n" event={"ID":"9d176a6b-c22a-4f6a-9551-1dbd25928cf8","Type":"ContainerStarted","Data":"70093b82a6e8a74c942202cf4b44ce16e8778421c550b4c8522bb1e6cb67942b"} Sep 07 03:49:10.427669 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.427624 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" event={"ID":"8b8f75fd-b50c-461a-9f2f-d8433861d39a","Type":"ContainerStarted","Data":"66e51cfda78422776a780d9d227f7afc1e7e96ec61cb448fce656c086e70a7fb"} Sep 07 03:49:10.429358 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.429329 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rh982" event={"ID":"08336dd4-dd4c-47e3-ba8d-531ca667a0b5","Type":"ContainerStarted","Data":"47371d770bca67038b06bc941aacca5b01a2bbfd021e32db6e59814f0aa1565a"} Sep 07 03:49:10.431186 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.431162 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7447dd5b74-p7pm5" event={"ID":"cc36ab08-3dc2-4011-b24a-4ee2635c4630","Type":"ContainerStarted","Data":"1ae11e9469904e0f91bd544dc72f3d92c67f5884a8ddbcb5c8213cb1464e8b6b"} Sep 07 03:49:10.431731 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.431201 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7447dd5b74-p7pm5" event={"ID":"cc36ab08-3dc2-4011-b24a-4ee2635c4630","Type":"ContainerStarted","Data":"64a22fe05e11ab60c0c7c3e35f9a828438e04aacffbb4ff720ae38b50c751554"} Sep 07 03:49:10.443825 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.442900 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" podStartSLOduration=68.442881963 podStartE2EDuration="1m8.442881963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:10.441460549 +0000 UTC m=+69.016326843" watchObservedRunningTime="2026-09-07 03:49:10.442881963 +0000 UTC m=+69.017748253" Sep 07 03:49:10.458698 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.458647 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sxp6n" podStartSLOduration=67.269306994 podStartE2EDuration="1m8.458634702s" totalImagesPullingTime="1.189327708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:10.458026498 +0000 UTC m=+69.032892789" watchObservedRunningTime="2026-09-07 03:49:10.458634702 +0000 UTC m=+69.033500994" Sep 07 03:49:10.482401 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.482348 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7447dd5b74-p7pm5" podStartSLOduration=61.482334921 podStartE2EDuration="1m1.482334921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:10.48156841 +0000 UTC m=+69.056434700" watchObservedRunningTime="2026-09-07 03:49:10.482334921 +0000 UTC m=+69.057201220" Sep 07 03:49:10.860901 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.860858 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:49:10.864471 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.864438 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:49:10.931339 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:10.931310 2555 scope.go:117] "RemoveContainer" containerID="329d8138d4635f4a559e3331f5633e9c9d3d847e6c40de9596546311c934be6f" Sep 07 03:49:11.438110 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:11.438036 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:49:11.440012 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:11.439823 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7447dd5b74-p7pm5" Sep 07 03:49:20.476839 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.476798 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" event={"ID":"8b8f75fd-b50c-461a-9f2f-d8433861d39a","Type":"ContainerStarted","Data":"6763ff71326da64741a91529c23703f3cff01cc2e848b76afde2019122d98051"} Sep 07 03:49:20.478840 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.478808 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rh982" event={"ID":"08336dd4-dd4c-47e3-ba8d-531ca667a0b5","Type":"ContainerStarted","Data":"39bf3b2942da4757574c030f8745cc9535fea0a018bd5c10ca86d4fb68b6fe53"} Sep 07 03:49:20.478972 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.478855 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rh982" event={"ID":"08336dd4-dd4c-47e3-ba8d-531ca667a0b5","Type":"ContainerStarted","Data":"bee6c70a8b0edf45392683da7fe00604c78f5c0932d1bbc2be6f530d5b883fca"} Sep 07 03:49:20.478972 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.478942 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rh982" Sep 07 03:49:20.480731 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.480713 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 03:49:20.480834 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.480817 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" event={"ID":"577e74b9-769c-48fa-b70f-e87314e9f179","Type":"ContainerStarted","Data":"f07923c7e6c3ac753bb08e89674bf5d993c0e5e23dfeef4b80e90167d8ad9425"} Sep 07 03:49:20.481079 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.481025 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:49:20.482422 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.482403 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gbx7k" event={"ID":"4a784083-f7eb-4508-9ef5-07ae8c51a973","Type":"ContainerStarted","Data":"4989d31fdb17f906b8f7bcdf4f91e6972d2ec6310c67f3af6540cb9d83bc5462"} Sep 07 03:49:20.484028 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.484007 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" event={"ID":"a15770ff-4db9-4dc4-8971-a7eeb7953137","Type":"ContainerStarted","Data":"f76544a94eb7aac22b352d6f5874dfa289869ebe9ee3ae486e8a1815b2c189d6"} Sep 07 03:49:20.484141 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.484068 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:20.485750 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.485717 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" event={"ID":"6c4c4546-b1dd-431c-a4d1-0144d4b698d7","Type":"ContainerStarted","Data":"b020901b7d5a8db749ecbb59798f62abd24414808be495b6f88cd158d49e119e"} Sep 07 03:49:20.486077 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.486006 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" Sep 07 03:49:20.488864 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.488838 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" Sep 07 03:49:20.490832 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.490811 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" event={"ID":"d26005b5-a488-4b37-bb20-a11934c65b28","Type":"ContainerStarted","Data":"c16bcbc41c1439f5321b39d915ceea50d48bb04701046ea00f2776f7a3551373"} Sep 07 03:49:20.490940 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.490844 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" event={"ID":"d26005b5-a488-4b37-bb20-a11934c65b28","Type":"ContainerStarted","Data":"076d7a37e203dfffd75b204e65f2eb5bbbd81b7cb9918ec1075299ad8aa9d4ff"} Sep 07 03:49:20.490940 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.490862 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" event={"ID":"d26005b5-a488-4b37-bb20-a11934c65b28","Type":"ContainerStarted","Data":"7391314240a5a61b112f2495c2827d7bb95e556e6fda32a01f0350b4637f54f3"} Sep 07 03:49:20.495584 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.495543 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5qh6c" podStartSLOduration=62.697636423 podStartE2EDuration="1m11.495529395s" totalImagesPullingTime="8.797892972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:20.494306286 +0000 UTC m=+79.069172576" watchObservedRunningTime="2026-09-07 03:49:20.495529395 +0000 UTC m=+79.070395684" Sep 07 03:49:20.517645 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.517605 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d67ccdcb9-nhgsq" podStartSLOduration=1.715563259 podStartE2EDuration="11.517591204s" totalImagesPullingTime="9.802027945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:20.516291684 +0000 UTC m=+79.091157976" watchObservedRunningTime="2026-09-07 03:49:20.517591204 +0000 UTC m=+79.092457494" Sep 07 03:49:20.542580 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.542531 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rh982" podStartSLOduration=35.743138939 podStartE2EDuration="44.542513637s" totalImagesPullingTime="8.799374698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:20.54043908 +0000 UTC m=+79.115305369" watchObservedRunningTime="2026-09-07 03:49:20.542513637 +0000 UTC m=+79.117379924" Sep 07 03:49:20.566509 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.566457 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84d98fc695-pqcqj" podStartSLOduration=2.267901164 podStartE2EDuration="11.566441901s" totalImagesPullingTime="9.298540737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:20.566027313 +0000 UTC m=+79.140893604" watchObservedRunningTime="2026-09-07 03:49:20.566441901 +0000 UTC m=+79.141308192" Sep 07 03:49:20.590693 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.590647 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gbx7k" podStartSLOduration=67.282319974 podStartE2EDuration="1m18.59063087s" totalImagesPullingTime="11.308310896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 03:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:20.588577829 +0000 UTC m=+79.163444118" watchObservedRunningTime="2026-09-07 03:49:20.59063087 +0000 UTC m=+79.165497160" Sep 07 03:49:20.606178 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.606142 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-zvrg9" podStartSLOduration=56.788529365 podStartE2EDuration="1m4.606133049s" totalImagesPullingTime="7.817603684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:20.605923911 +0000 UTC m=+79.180790204" watchObservedRunningTime="2026-09-07 03:49:20.606133049 +0000 UTC m=+79.180999337" Sep 07 03:49:20.624005 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:20.623968 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78889d8498-z5w76" podStartSLOduration=2.3543410209999998 podStartE2EDuration="11.623952536s" totalImagesPullingTime="9.269611515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:20.62307466 +0000 UTC m=+79.197940947" watchObservedRunningTime="2026-09-07 03:49:20.623952536 +0000 UTC m=+79.198818827" Sep 07 03:49:27.673644 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.673520 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f6k55"] Sep 07 03:49:27.680118 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.680095 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.684320 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.684300 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qtb9h\"" Sep 07 03:49:27.684857 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.684837 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 07 03:49:27.685158 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.684884 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 07 03:49:27.685280 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.684896 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 07 03:49:27.685395 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.684896 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 07 03:49:27.874412 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.874370 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.874603 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.874457 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c18d70f7-9e5a-4ab5-902a-c19904a894f7-sys\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.874603 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.874495 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-wtmp\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.874603 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.874559 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c18d70f7-9e5a-4ab5-902a-c19904a894f7-metrics-client-ca\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.874765 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.874602 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-textfile\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.874765 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.874622 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.874765 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.874640 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42q7r\" (UniqueName: \"kubernetes.io/projected/c18d70f7-9e5a-4ab5-902a-c19904a894f7-kube-api-access-42q7r\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.874765 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.874659 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-tls\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.874765 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.874726 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c18d70f7-9e5a-4ab5-902a-c19904a894f7-root\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.975711 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.975677 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-wtmp\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.975886 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.975725 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c18d70f7-9e5a-4ab5-902a-c19904a894f7-metrics-client-ca\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.975886 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.975870 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-textfile\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.975995 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.975915 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.975995 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.975939 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-wtmp\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.975995 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.975945 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42q7r\" (UniqueName: \"kubernetes.io/projected/c18d70f7-9e5a-4ab5-902a-c19904a894f7-kube-api-access-42q7r\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.976172 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.976000 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-tls\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.976172 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.976031 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c18d70f7-9e5a-4ab5-902a-c19904a894f7-root\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.976172 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.976092 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.976331 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.976181 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c18d70f7-9e5a-4ab5-902a-c19904a894f7-sys\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.976331 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.976210 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-textfile\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.976331 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.976267 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c18d70f7-9e5a-4ab5-902a-c19904a894f7-sys\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.976331 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:49:27.976294 2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 07 03:49:27.976331 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.976312 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c18d70f7-9e5a-4ab5-902a-c19904a894f7-root\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.976523 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:49:27.976357 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-tls podName:c18d70f7-9e5a-4ab5-902a-c19904a894f7 nodeName:}" failed. No retries permitted until 2026-09-07 03:49:28.476339091 +0000 UTC m=+87.051205364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-tls") pod "node-exporter-f6k55" (UID: "c18d70f7-9e5a-4ab5-902a-c19904a894f7") : secret "node-exporter-tls" not found Sep 07 03:49:27.976523 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.976365 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c18d70f7-9e5a-4ab5-902a-c19904a894f7-metrics-client-ca\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.976699 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.976679 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.978291 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.978270 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:27.984295 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:27.984275 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42q7r\" (UniqueName: \"kubernetes.io/projected/c18d70f7-9e5a-4ab5-902a-c19904a894f7-kube-api-access-42q7r\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:28.480382 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:28.480349 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-tls\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:28.482667 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:28.482642 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c18d70f7-9e5a-4ab5-902a-c19904a894f7-node-exporter-tls\") pod \"node-exporter-f6k55\" (UID: \"c18d70f7-9e5a-4ab5-902a-c19904a894f7\") " pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:28.593030 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:28.592999 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f6k55" Sep 07 03:49:28.608643 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:49:28.608617 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18d70f7_9e5a_4ab5_902a_c19904a894f7.slice/crio-8c2e050757dcdb2639339192d36bae807f7b79966558cb33b2dfbef166e3619f WatchSource:0}: Error finding container 8c2e050757dcdb2639339192d36bae807f7b79966558cb33b2dfbef166e3619f: Status 404 returned error can't find the container with id 8c2e050757dcdb2639339192d36bae807f7b79966558cb33b2dfbef166e3619f Sep 07 03:49:28.899728 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:28.899649 2555 patch_prober.go:28] interesting pod/image-registry-7b7f8788ff-2zcb2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 07 03:49:28.899728 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:28.899700 2555 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" podUID="66017c83-f8c1-4280-aefa-499d9861a68f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 03:49:29.525778 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:29.525754 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f6k55" event={"ID":"c18d70f7-9e5a-4ab5-902a-c19904a894f7","Type":"ContainerStarted","Data":"8c2e050757dcdb2639339192d36bae807f7b79966558cb33b2dfbef166e3619f"} Sep 07 03:49:30.498218 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:30.498185 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rh982" Sep 07 03:49:30.535369 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:30.535332 2555 generic.go:356] "Generic (PLEG): container finished" podID="c18d70f7-9e5a-4ab5-902a-c19904a894f7" containerID="98c65f324938644b1201df35035cc75a0f55cf112d6e79ea79890cb4ba2b1201" exitCode=0 Sep 07 03:49:30.535548 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:30.535418 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f6k55" event={"ID":"c18d70f7-9e5a-4ab5-902a-c19904a894f7","Type":"ContainerDied","Data":"98c65f324938644b1201df35035cc75a0f55cf112d6e79ea79890cb4ba2b1201"} Sep 07 03:49:31.442089 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:31.442040 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:49:31.542350 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:31.542318 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f6k55" event={"ID":"c18d70f7-9e5a-4ab5-902a-c19904a894f7","Type":"ContainerStarted","Data":"5263452af97c16266c5c86991c25726cb0bc69f6f7adc24f15aba8a944cd2692"} Sep 07 03:49:31.542705 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:31.542368 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f6k55" event={"ID":"c18d70f7-9e5a-4ab5-902a-c19904a894f7","Type":"ContainerStarted","Data":"f4a2b0db91c7a3d23256bc72263c2607360f8a4c14a802e9263f03259537f138"} Sep 07 03:49:31.559465 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:31.559440 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b7f8788ff-2zcb2"] Sep 07 03:49:31.572768 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:31.572713 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f6k55" podStartSLOduration=3.647126349 podStartE2EDuration="4.572692568s" totalImagesPullingTime="864.897845ms" totalInitContainerRuntime="60.668374ms" isStatefulPod=true podCreationTimestamp="2026-09-07 03:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:49:31.571580698 +0000 UTC m=+90.146446988" watchObservedRunningTime="2026-09-07 03:49:31.572692568 +0000 UTC m=+90.147558858" Sep 07 03:49:39.411103 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:39.411048 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x6689" Sep 07 03:49:56.582761 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.582723 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" podUID="66017c83-f8c1-4280-aefa-499d9861a68f" containerName="registry" containerID="cri-o://3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb" gracePeriod=30 Sep 07 03:49:56.835401 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.835354 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:49:56.893715 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.893686 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-trusted-ca\") pod \"66017c83-f8c1-4280-aefa-499d9861a68f\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " Sep 07 03:49:56.893838 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.893721 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66017c83-f8c1-4280-aefa-499d9861a68f-ca-trust-extracted\") pod \"66017c83-f8c1-4280-aefa-499d9861a68f\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " Sep 07 03:49:56.893838 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.893736 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-registry-certificates\") pod \"66017c83-f8c1-4280-aefa-499d9861a68f\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " Sep 07 03:49:56.893838 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.893774 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-bound-sa-token\") pod \"66017c83-f8c1-4280-aefa-499d9861a68f\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " Sep 07 03:49:56.893838 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.893806 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") pod \"66017c83-f8c1-4280-aefa-499d9861a68f\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " Sep 07 03:49:56.894008 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.893838 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6drv\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-kube-api-access-z6drv\") pod \"66017c83-f8c1-4280-aefa-499d9861a68f\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " Sep 07 03:49:56.894085 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.894003 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-image-registry-private-configuration\") pod \"66017c83-f8c1-4280-aefa-499d9861a68f\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " Sep 07 03:49:56.894144 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.894094 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-installation-pull-secrets\") pod \"66017c83-f8c1-4280-aefa-499d9861a68f\" (UID: \"66017c83-f8c1-4280-aefa-499d9861a68f\") " Sep 07 03:49:56.894203 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.894178 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "66017c83-f8c1-4280-aefa-499d9861a68f" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 03:49:56.894403 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.894367 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-trusted-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:49:56.894540 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.894516 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "66017c83-f8c1-4280-aefa-499d9861a68f" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 03:49:56.896494 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.896447 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "66017c83-f8c1-4280-aefa-499d9861a68f" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 03:49:56.896587 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.896513 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-kube-api-access-z6drv" (OuterVolumeSpecName: "kube-api-access-z6drv") pod "66017c83-f8c1-4280-aefa-499d9861a68f" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f"). InnerVolumeSpecName "kube-api-access-z6drv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:49:56.896671 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.896641 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "66017c83-f8c1-4280-aefa-499d9861a68f" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:49:56.896715 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.896644 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "66017c83-f8c1-4280-aefa-499d9861a68f" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 03:49:56.896753 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.896709 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "66017c83-f8c1-4280-aefa-499d9861a68f" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:49:56.905123 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.905097 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66017c83-f8c1-4280-aefa-499d9861a68f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "66017c83-f8c1-4280-aefa-499d9861a68f" (UID: "66017c83-f8c1-4280-aefa-499d9861a68f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 03:49:56.995366 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.995331 2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66017c83-f8c1-4280-aefa-499d9861a68f-ca-trust-extracted\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:49:56.995366 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.995362 2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66017c83-f8c1-4280-aefa-499d9861a68f-registry-certificates\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:49:56.995366 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.995372 2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-bound-sa-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:49:56.995518 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.995381 2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-registry-tls\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:49:56.995518 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.995389 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6drv\" (UniqueName: \"kubernetes.io/projected/66017c83-f8c1-4280-aefa-499d9861a68f-kube-api-access-z6drv\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:49:56.995518 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.995399 2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-image-registry-private-configuration\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:49:56.995518 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:56.995408 2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66017c83-f8c1-4280-aefa-499d9861a68f-installation-pull-secrets\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:49:57.642231 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:57.642190 2555 generic.go:356] "Generic (PLEG): container finished" podID="66017c83-f8c1-4280-aefa-499d9861a68f" containerID="3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb" exitCode=0 Sep 07 03:49:57.642609 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:57.642253 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" Sep 07 03:49:57.642609 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:57.642279 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" event={"ID":"66017c83-f8c1-4280-aefa-499d9861a68f","Type":"ContainerDied","Data":"3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb"} Sep 07 03:49:57.642609 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:57.642357 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b7f8788ff-2zcb2" event={"ID":"66017c83-f8c1-4280-aefa-499d9861a68f","Type":"ContainerDied","Data":"4684ef973366e7f25dff7f16287bff139b876d5906dc01b24eab686f78589d82"} Sep 07 03:49:57.642609 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:57.642378 2555 scope.go:117] "RemoveContainer" containerID="3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb" Sep 07 03:49:57.651456 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:57.651436 2555 scope.go:117] "RemoveContainer" containerID="3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb" Sep 07 03:49:57.651730 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:49:57.651710 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb\": container with ID starting with 3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb not found: ID does not exist" containerID="3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb" Sep 07 03:49:57.651795 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:57.651738 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb"} err="failed to get container status \"3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb\": rpc error: code = NotFound desc = could not find container \"3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb\": container with ID starting with 3d0e08cd631e011a647782b958f74ce755db51aaea56b9a61394b06fb5ef81eb not found: ID does not exist" Sep 07 03:49:57.664051 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:57.664030 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b7f8788ff-2zcb2"] Sep 07 03:49:57.671793 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:57.671774 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b7f8788ff-2zcb2"] Sep 07 03:49:57.935355 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:49:57.935291 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66017c83-f8c1-4280-aefa-499d9861a68f" path="/var/lib/kubelet/pods/66017c83-f8c1-4280-aefa-499d9861a68f/volumes" Sep 07 03:50:06.678282 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:06.678249 2555 generic.go:356] "Generic (PLEG): container finished" podID="ed5e0dcd-3686-4f2d-8710-36f47ade6997" containerID="e9f4381877e0651fb213a6a515573bf50e512a700b160fa617f3556364bf25c1" exitCode=0 Sep 07 03:50:06.678616 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:06.678320 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" event={"ID":"ed5e0dcd-3686-4f2d-8710-36f47ade6997","Type":"ContainerDied","Data":"e9f4381877e0651fb213a6a515573bf50e512a700b160fa617f3556364bf25c1"} Sep 07 03:50:06.678662 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:06.678638 2555 scope.go:117] "RemoveContainer" containerID="e9f4381877e0651fb213a6a515573bf50e512a700b160fa617f3556364bf25c1" Sep 07 03:50:07.683891 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:07.683802 2555 generic.go:356] "Generic (PLEG): container finished" podID="4e43e9f6-5add-434b-89cc-d9db04f24dc7" containerID="87f75c27cc448b4fc879e342100c58d1bdd427bb62e9fb70ad847ac7719c418a" exitCode=0 Sep 07 03:50:07.683891 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:07.683876 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" event={"ID":"4e43e9f6-5add-434b-89cc-d9db04f24dc7","Type":"ContainerDied","Data":"87f75c27cc448b4fc879e342100c58d1bdd427bb62e9fb70ad847ac7719c418a"} Sep 07 03:50:07.684391 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:07.684227 2555 scope.go:117] "RemoveContainer" containerID="87f75c27cc448b4fc879e342100c58d1bdd427bb62e9fb70ad847ac7719c418a" Sep 07 03:50:07.685868 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:07.685850 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-bml8f" event={"ID":"ed5e0dcd-3686-4f2d-8710-36f47ade6997","Type":"ContainerStarted","Data":"6e5f439a61f36da8f54b6c5a200e5298fe74b130a0b221e25c837af52ccc7451"} Sep 07 03:50:08.691861 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:08.691813 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rmznn" event={"ID":"4e43e9f6-5add-434b-89cc-d9db04f24dc7","Type":"ContainerStarted","Data":"2513871c239a9e875d8ad5e56f7fea6e4c4c4aee89f81ca831e1f38258208dc1"} Sep 07 03:50:16.722445 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:16.722415 2555 generic.go:356] "Generic (PLEG): container finished" podID="4a5acee8-01de-477c-a111-d0f3ea9b2afb" containerID="d99a70c3b6966c67411d71ae60d7b119f4bd27780730dcc86acd2c5c4a6f625a" exitCode=0 Sep 07 03:50:16.722737 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:16.722462 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" event={"ID":"4a5acee8-01de-477c-a111-d0f3ea9b2afb","Type":"ContainerDied","Data":"d99a70c3b6966c67411d71ae60d7b119f4bd27780730dcc86acd2c5c4a6f625a"} Sep 07 03:50:16.722793 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:16.722745 2555 scope.go:117] "RemoveContainer" containerID="d99a70c3b6966c67411d71ae60d7b119f4bd27780730dcc86acd2c5c4a6f625a" Sep 07 03:50:17.727509 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:50:17.727474 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5m95s" event={"ID":"4a5acee8-01de-477c-a111-d0f3ea9b2afb","Type":"ContainerStarted","Data":"54810cab3795c4f151c3b793c1e0de696fecfc551314ddcbcfba722b6f6561a1"} Sep 07 03:52:56.661682 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.661648 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs"] Sep 07 03:52:56.662140 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.661925 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66017c83-f8c1-4280-aefa-499d9861a68f" containerName="registry" Sep 07 03:52:56.662140 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.661936 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66017c83-f8c1-4280-aefa-499d9861a68f" containerName="registry" Sep 07 03:52:56.662140 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.662004 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="66017c83-f8c1-4280-aefa-499d9861a68f" containerName="registry" Sep 07 03:52:56.664734 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.664717 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.675035 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.675016 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 07 03:52:56.675976 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.675959 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7q6jm\"" Sep 07 03:52:56.676258 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.676243 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 07 03:52:56.678667 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.678649 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 07 03:52:56.679038 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.679022 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 07 03:52:56.699817 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.699795 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs"] Sep 07 03:52:56.762242 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.762213 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e6dc622d-3312-4719-956d-401de581ccc7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.762392 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.762275 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e6dc622d-3312-4719-956d-401de581ccc7-manifests\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.762392 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.762373 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s6jb\" (UniqueName: \"kubernetes.io/projected/e6dc622d-3312-4719-956d-401de581ccc7-kube-api-access-5s6jb\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.762508 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.762424 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6dc622d-3312-4719-956d-401de581ccc7-webhook-cert\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.762508 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.762475 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6dc622d-3312-4719-956d-401de581ccc7-tmp\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.863598 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.863567 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s6jb\" (UniqueName: \"kubernetes.io/projected/e6dc622d-3312-4719-956d-401de581ccc7-kube-api-access-5s6jb\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.863738 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.863603 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6dc622d-3312-4719-956d-401de581ccc7-webhook-cert\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.863738 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.863628 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6dc622d-3312-4719-956d-401de581ccc7-tmp\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.863738 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.863648 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e6dc622d-3312-4719-956d-401de581ccc7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.863738 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.863687 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e6dc622d-3312-4719-956d-401de581ccc7-manifests\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.864083 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.864031 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e6dc622d-3312-4719-956d-401de581ccc7-manifests\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.866001 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.865977 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6dc622d-3312-4719-956d-401de581ccc7-tmp\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.866113 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.866078 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6dc622d-3312-4719-956d-401de581ccc7-webhook-cert\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.866171 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.866128 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e6dc622d-3312-4719-956d-401de581ccc7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.872118 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.872098 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s6jb\" (UniqueName: \"kubernetes.io/projected/e6dc622d-3312-4719-956d-401de581ccc7-kube-api-access-5s6jb\") pod \"opendatahub-operator-controller-manager-756ff4ddd-r5gvs\" (UID: \"e6dc622d-3312-4719-956d-401de581ccc7\") " pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:56.973877 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:56.973848 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:52:57.251634 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:57.251559 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs"] Sep 07 03:52:57.255362 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:52:57.255338 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6dc622d_3312_4719_956d_401de581ccc7.slice/crio-fb3f1c393f0b1f6cad606a9ac8da1ea1fab9683517dd99cf9d4f1d9f0646d801 WatchSource:0}: Error finding container fb3f1c393f0b1f6cad606a9ac8da1ea1fab9683517dd99cf9d4f1d9f0646d801: Status 404 returned error can't find the container with id fb3f1c393f0b1f6cad606a9ac8da1ea1fab9683517dd99cf9d4f1d9f0646d801 Sep 07 03:52:57.320269 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:52:57.320241 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" event={"ID":"e6dc622d-3312-4719-956d-401de581ccc7","Type":"ContainerStarted","Data":"fb3f1c393f0b1f6cad606a9ac8da1ea1fab9683517dd99cf9d4f1d9f0646d801"} Sep 07 03:53:00.336042 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:00.336009 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" event={"ID":"e6dc622d-3312-4719-956d-401de581ccc7","Type":"ContainerStarted","Data":"ace45f7ebfeb7adc6c22bc9a4726badba0f80267b2d75dcc7323b7511270556e"} Sep 07 03:53:01.857278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:01.857250 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 03:53:01.857733 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:01.857250 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 03:53:01.869107 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:01.869085 2555 kubelet.go:1648] "Image garbage collection succeeded" Sep 07 03:53:02.346726 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:02.346692 2555 generic.go:356] "Generic (PLEG): container finished" podID="e6dc622d-3312-4719-956d-401de581ccc7" containerID="ace45f7ebfeb7adc6c22bc9a4726badba0f80267b2d75dcc7323b7511270556e" exitCode=0 Sep 07 03:53:02.348849 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:02.346754 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" event={"ID":"e6dc622d-3312-4719-956d-401de581ccc7","Type":"ContainerDied","Data":"ace45f7ebfeb7adc6c22bc9a4726badba0f80267b2d75dcc7323b7511270556e"} Sep 07 03:53:03.353200 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:03.353167 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" event={"ID":"e6dc622d-3312-4719-956d-401de581ccc7","Type":"ContainerStarted","Data":"1eb0f5576e16e98c61c37a06865eb82f7ee76e265da3bbf234e5c3a940b0ba16"} Sep 07 03:53:03.353651 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:03.353390 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:53:03.376516 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:03.376468 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" podStartSLOduration=3.054528481 podStartE2EDuration="7.376452675s" totalImagesPullingTime="3.232707777s" totalInitContainerRuntime="1.089216417s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:52:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:53:03.375220425 +0000 UTC m=+301.950086714" watchObservedRunningTime="2026-09-07 03:53:03.376452675 +0000 UTC m=+301.951318964" Sep 07 03:53:14.360221 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:14.360189 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-756ff4ddd-r5gvs" Sep 07 03:53:17.517314 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.517281 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2"] Sep 07 03:53:17.531315 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.531290 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:17.531621 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.531593 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2"] Sep 07 03:53:17.534001 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.533976 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 07 03:53:17.534126 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.533980 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-d46lm\"" Sep 07 03:53:17.534126 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.534015 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 07 03:53:17.625200 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.625176 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6fh\" (UniqueName: \"kubernetes.io/projected/b78d4100-aaa6-4b75-9804-2a7d5faa86c3-kube-api-access-2h6fh\") pod \"kube-auth-proxy-59d7d88d64-pvnz2\" (UID: \"b78d4100-aaa6-4b75-9804-2a7d5faa86c3\") " pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:17.625297 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.625218 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b78d4100-aaa6-4b75-9804-2a7d5faa86c3-tls-certs\") pod \"kube-auth-proxy-59d7d88d64-pvnz2\" (UID: \"b78d4100-aaa6-4b75-9804-2a7d5faa86c3\") " pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:17.625297 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.625243 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b78d4100-aaa6-4b75-9804-2a7d5faa86c3-tmp\") pod \"kube-auth-proxy-59d7d88d64-pvnz2\" (UID: \"b78d4100-aaa6-4b75-9804-2a7d5faa86c3\") " pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:17.726461 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.726428 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6fh\" (UniqueName: \"kubernetes.io/projected/b78d4100-aaa6-4b75-9804-2a7d5faa86c3-kube-api-access-2h6fh\") pod \"kube-auth-proxy-59d7d88d64-pvnz2\" (UID: \"b78d4100-aaa6-4b75-9804-2a7d5faa86c3\") " pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:17.726589 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.726479 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b78d4100-aaa6-4b75-9804-2a7d5faa86c3-tls-certs\") pod \"kube-auth-proxy-59d7d88d64-pvnz2\" (UID: \"b78d4100-aaa6-4b75-9804-2a7d5faa86c3\") " pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:17.726589 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.726502 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b78d4100-aaa6-4b75-9804-2a7d5faa86c3-tmp\") pod \"kube-auth-proxy-59d7d88d64-pvnz2\" (UID: \"b78d4100-aaa6-4b75-9804-2a7d5faa86c3\") " pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:17.728775 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.728753 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b78d4100-aaa6-4b75-9804-2a7d5faa86c3-tmp\") pod \"kube-auth-proxy-59d7d88d64-pvnz2\" (UID: \"b78d4100-aaa6-4b75-9804-2a7d5faa86c3\") " pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:17.728875 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.728857 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b78d4100-aaa6-4b75-9804-2a7d5faa86c3-tls-certs\") pod \"kube-auth-proxy-59d7d88d64-pvnz2\" (UID: \"b78d4100-aaa6-4b75-9804-2a7d5faa86c3\") " pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:17.734941 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.734921 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6fh\" (UniqueName: \"kubernetes.io/projected/b78d4100-aaa6-4b75-9804-2a7d5faa86c3-kube-api-access-2h6fh\") pod \"kube-auth-proxy-59d7d88d64-pvnz2\" (UID: \"b78d4100-aaa6-4b75-9804-2a7d5faa86c3\") " pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:17.841932 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:17.841878 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" Sep 07 03:53:18.112878 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.112809 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2"] Sep 07 03:53:18.115703 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:53:18.115671 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78d4100_aaa6_4b75_9804_2a7d5faa86c3.slice/crio-0fab71b2abb2861132849522eb9a60b686434f76454a9f34bf218dc8bd2a4c84 WatchSource:0}: Error finding container 0fab71b2abb2861132849522eb9a60b686434f76454a9f34bf218dc8bd2a4c84: Status 404 returned error can't find the container with id 0fab71b2abb2861132849522eb9a60b686434f76454a9f34bf218dc8bd2a4c84 Sep 07 03:53:18.117716 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.117694 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 03:53:18.416827 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.416742 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" event={"ID":"b78d4100-aaa6-4b75-9804-2a7d5faa86c3","Type":"ContainerStarted","Data":"0fab71b2abb2861132849522eb9a60b686434f76454a9f34bf218dc8bd2a4c84"} Sep 07 03:53:18.592860 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.590200 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw"] Sep 07 03:53:18.596417 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.596388 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.598697 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.598673 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 07 03:53:18.598823 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.598679 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 07 03:53:18.598823 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.598680 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 07 03:53:18.599212 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.599194 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 07 03:53:18.599286 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.599266 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6sc2g\"" Sep 07 03:53:18.604684 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.604662 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw"] Sep 07 03:53:18.734346 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.734308 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.734346 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.734352 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.734552 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.734381 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7827b2d1-c6cb-4b62-afff-6f22ff75e258-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.734552 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.734410 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.734552 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.734442 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwgvb\" (UniqueName: \"kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-kube-api-access-lwgvb\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.734552 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.734504 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.734678 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.734553 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.835430 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.835350 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.835430 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.835416 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.835676 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.835472 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.835676 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.835496 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.835676 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.835533 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7827b2d1-c6cb-4b62-afff-6f22ff75e258-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.835676 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.835573 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.835676 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.835600 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwgvb\" (UniqueName: \"kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-kube-api-access-lwgvb\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.836774 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.836709 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.839014 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.838992 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.839680 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.839642 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7827b2d1-c6cb-4b62-afff-6f22ff75e258-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.839853 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.839829 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.839945 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.839919 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.843752 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.843710 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwgvb\" (UniqueName: \"kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-kube-api-access-lwgvb\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.844291 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.844268 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-t8dqw\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:18.906033 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:18.906004 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:19.197230 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:19.196993 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw"] Sep 07 03:53:19.200599 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:53:19.200476 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7827b2d1_c6cb_4b62_afff_6f22ff75e258.slice/crio-8005b10ac2a05a93a590e8a96156ef3cd04be4f34886c24c68c19c41dae12e81 WatchSource:0}: Error finding container 8005b10ac2a05a93a590e8a96156ef3cd04be4f34886c24c68c19c41dae12e81: Status 404 returned error can't find the container with id 8005b10ac2a05a93a590e8a96156ef3cd04be4f34886c24c68c19c41dae12e81 Sep 07 03:53:19.424524 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:19.424483 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" event={"ID":"7827b2d1-c6cb-4b62-afff-6f22ff75e258","Type":"ContainerStarted","Data":"8005b10ac2a05a93a590e8a96156ef3cd04be4f34886c24c68c19c41dae12e81"} Sep 07 03:53:21.438751 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:21.437999 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" event={"ID":"b78d4100-aaa6-4b75-9804-2a7d5faa86c3","Type":"ContainerStarted","Data":"e6dc8017fcf6ec4801ba9a0fa9311ec371318b959af22775d5f9eaf76a921456"} Sep 07 03:53:21.457998 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:21.457951 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59d7d88d64-pvnz2" podStartSLOduration=1.725130814 podStartE2EDuration="4.457931728s" totalImagesPullingTime="2.732800914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:53:21.456186033 +0000 UTC m=+320.031052330" watchObservedRunningTime="2026-09-07 03:53:21.457931728 +0000 UTC m=+320.032798018" Sep 07 03:53:22.665008 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:22.664969 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 03:53:22.665252 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:22.665073 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 03:53:23.450137 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:23.450085 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" event={"ID":"7827b2d1-c6cb-4b62-afff-6f22ff75e258","Type":"ContainerStarted","Data":"4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0"} Sep 07 03:53:23.450364 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:23.450186 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:23.451929 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:23.451898 2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-t8dqw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 03:53:23.452094 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:23.451947 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" podUID="7827b2d1-c6cb-4b62-afff-6f22ff75e258" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 03:53:23.471791 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:23.471742 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" podStartSLOduration=2.010468285 podStartE2EDuration="5.471727542s" totalImagesPullingTime="3.461259257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:53:23.4700954 +0000 UTC m=+322.044961690" watchObservedRunningTime="2026-09-07 03:53:23.471727542 +0000 UTC m=+322.046593831" Sep 07 03:53:24.455984 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:24.455956 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:37.493580 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:37.493543 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-44pf7"] Sep 07 03:53:37.497087 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:37.497046 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:53:37.510985 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:37.510963 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 07 03:53:37.511224 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:37.511206 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-56k8j\"" Sep 07 03:53:37.530637 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:37.530608 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-44pf7"] Sep 07 03:53:37.580774 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:37.580735 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58sbk\" (UniqueName: \"kubernetes.io/projected/91c7d218-7958-4f12-9ddc-b06d91c091b8-kube-api-access-58sbk\") pod \"kserve-controller-manager-d6f4b65b8-44pf7\" (UID: \"91c7d218-7958-4f12-9ddc-b06d91c091b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:53:37.580944 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:37.580786 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91c7d218-7958-4f12-9ddc-b06d91c091b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-44pf7\" (UID: \"91c7d218-7958-4f12-9ddc-b06d91c091b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:53:37.682134 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:37.682096 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58sbk\" (UniqueName: \"kubernetes.io/projected/91c7d218-7958-4f12-9ddc-b06d91c091b8-kube-api-access-58sbk\") pod \"kserve-controller-manager-d6f4b65b8-44pf7\" (UID: \"91c7d218-7958-4f12-9ddc-b06d91c091b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:53:37.682325 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:37.682142 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91c7d218-7958-4f12-9ddc-b06d91c091b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-44pf7\" (UID: \"91c7d218-7958-4f12-9ddc-b06d91c091b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:53:37.682325 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:53:37.682270 2555 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 07 03:53:37.682444 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:53:37.682336 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c7d218-7958-4f12-9ddc-b06d91c091b8-cert podName:91c7d218-7958-4f12-9ddc-b06d91c091b8 nodeName:}" failed. No retries permitted until 2026-09-07 03:53:38.182313412 +0000 UTC m=+336.757179679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/91c7d218-7958-4f12-9ddc-b06d91c091b8-cert") pod "kserve-controller-manager-d6f4b65b8-44pf7" (UID: "91c7d218-7958-4f12-9ddc-b06d91c091b8") : secret "kserve-webhook-server-cert" not found Sep 07 03:53:37.699409 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:37.699379 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58sbk\" (UniqueName: \"kubernetes.io/projected/91c7d218-7958-4f12-9ddc-b06d91c091b8-kube-api-access-58sbk\") pod \"kserve-controller-manager-d6f4b65b8-44pf7\" (UID: \"91c7d218-7958-4f12-9ddc-b06d91c091b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:53:38.185852 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:38.185806 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91c7d218-7958-4f12-9ddc-b06d91c091b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-44pf7\" (UID: \"91c7d218-7958-4f12-9ddc-b06d91c091b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:53:38.188174 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:38.188154 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91c7d218-7958-4f12-9ddc-b06d91c091b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-44pf7\" (UID: \"91c7d218-7958-4f12-9ddc-b06d91c091b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:53:38.407302 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:38.407269 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:53:38.674758 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:38.674645 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-44pf7"] Sep 07 03:53:38.677358 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:53:38.677330 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c7d218_7958_4f12_9ddc_b06d91c091b8.slice/crio-d69f0ec231f8d897c362ab5bee6a03f508e33d791de2ddfd257204bc1bb07920 WatchSource:0}: Error finding container d69f0ec231f8d897c362ab5bee6a03f508e33d791de2ddfd257204bc1bb07920: Status 404 returned error can't find the container with id d69f0ec231f8d897c362ab5bee6a03f508e33d791de2ddfd257204bc1bb07920 Sep 07 03:53:39.521748 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:39.521704 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" event={"ID":"91c7d218-7958-4f12-9ddc-b06d91c091b8","Type":"ContainerStarted","Data":"d69f0ec231f8d897c362ab5bee6a03f508e33d791de2ddfd257204bc1bb07920"} Sep 07 03:53:39.797660 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:39.797575 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw"] Sep 07 03:53:39.798122 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:39.797762 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" podUID="7827b2d1-c6cb-4b62-afff-6f22ff75e258" containerName="discovery" containerID="cri-o://4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0" gracePeriod=30 Sep 07 03:53:40.056179 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.056122 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:40.101539 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.101508 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwgvb\" (UniqueName: \"kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-kube-api-access-lwgvb\") pod \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " Sep 07 03:53:40.101700 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.101555 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-cacerts\") pod \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " Sep 07 03:53:40.101700 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.101591 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-token\") pod \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " Sep 07 03:53:40.101700 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.101619 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7827b2d1-c6cb-4b62-afff-6f22ff75e258-local-certs\") pod \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " Sep 07 03:53:40.101700 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.101641 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-dns-cert\") pod \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " Sep 07 03:53:40.101700 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.101673 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-kubeconfig\") pod \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " Sep 07 03:53:40.101976 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.101746 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-ca-configmap\") pod \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\" (UID: \"7827b2d1-c6cb-4b62-afff-6f22ff75e258\") " Sep 07 03:53:40.102413 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.102374 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7827b2d1-c6cb-4b62-afff-6f22ff75e258" (UID: "7827b2d1-c6cb-4b62-afff-6f22ff75e258"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 03:53:40.104683 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.104640 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7827b2d1-c6cb-4b62-afff-6f22ff75e258" (UID: "7827b2d1-c6cb-4b62-afff-6f22ff75e258"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 03:53:40.105200 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.105161 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7827b2d1-c6cb-4b62-afff-6f22ff75e258-local-certs" (OuterVolumeSpecName: "local-certs") pod "7827b2d1-c6cb-4b62-afff-6f22ff75e258" (UID: "7827b2d1-c6cb-4b62-afff-6f22ff75e258"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 03:53:40.105289 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.105255 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7827b2d1-c6cb-4b62-afff-6f22ff75e258" (UID: "7827b2d1-c6cb-4b62-afff-6f22ff75e258"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 03:53:40.105344 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.105288 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-token" (OuterVolumeSpecName: "istio-token") pod "7827b2d1-c6cb-4b62-afff-6f22ff75e258" (UID: "7827b2d1-c6cb-4b62-afff-6f22ff75e258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:53:40.105344 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.105307 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-kube-api-access-lwgvb" (OuterVolumeSpecName: "kube-api-access-lwgvb") pod "7827b2d1-c6cb-4b62-afff-6f22ff75e258" (UID: "7827b2d1-c6cb-4b62-afff-6f22ff75e258"). InnerVolumeSpecName "kube-api-access-lwgvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:53:40.105814 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.105784 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-cacerts" (OuterVolumeSpecName: "cacerts") pod "7827b2d1-c6cb-4b62-afff-6f22ff75e258" (UID: "7827b2d1-c6cb-4b62-afff-6f22ff75e258"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 03:53:40.203124 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.203093 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwgvb\" (UniqueName: \"kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-kube-api-access-lwgvb\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:53:40.203124 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.203124 2555 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-cacerts\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:53:40.203327 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.203136 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:53:40.203327 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.203144 2555 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7827b2d1-c6cb-4b62-afff-6f22ff75e258-local-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:53:40.203327 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.203152 2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-dns-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:53:40.203327 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.203160 2555 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-kubeconfig\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:53:40.203327 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.203169 2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7827b2d1-c6cb-4b62-afff-6f22ff75e258-istio-csr-ca-configmap\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:53:40.528657 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.528616 2555 generic.go:356] "Generic (PLEG): container finished" podID="7827b2d1-c6cb-4b62-afff-6f22ff75e258" containerID="4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0" exitCode=0 Sep 07 03:53:40.528853 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.528688 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" event={"ID":"7827b2d1-c6cb-4b62-afff-6f22ff75e258","Type":"ContainerDied","Data":"4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0"} Sep 07 03:53:40.528853 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.528736 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" event={"ID":"7827b2d1-c6cb-4b62-afff-6f22ff75e258","Type":"ContainerDied","Data":"8005b10ac2a05a93a590e8a96156ef3cd04be4f34886c24c68c19c41dae12e81"} Sep 07 03:53:40.528853 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.528762 2555 scope.go:117] "RemoveContainer" containerID="4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0" Sep 07 03:53:40.529022 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.528882 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw" Sep 07 03:53:40.574579 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.574534 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw"] Sep 07 03:53:40.576277 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.576241 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-t8dqw"] Sep 07 03:53:40.697260 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.697237 2555 scope.go:117] "RemoveContainer" containerID="4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0" Sep 07 03:53:40.697596 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:53:40.697575 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0\": container with ID starting with 4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0 not found: ID does not exist" containerID="4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0" Sep 07 03:53:40.697645 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:40.697607 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0"} err="failed to get container status \"4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0\": rpc error: code = NotFound desc = could not find container \"4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0\": container with ID starting with 4025e6b179c5964b5d0749a2710b37dcfc57b43827675f5b16485acf78450fc0 not found: ID does not exist" Sep 07 03:53:41.536292 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:41.536257 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" event={"ID":"91c7d218-7958-4f12-9ddc-b06d91c091b8","Type":"ContainerStarted","Data":"3b2c43830a8d01adf46cbcea9620706e4c191d8f30eb27d29c74a5375d2cbc30"} Sep 07 03:53:41.536672 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:41.536414 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:53:41.553068 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:41.553023 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" podStartSLOduration=2.487781814 podStartE2EDuration="4.553012062s" totalImagesPullingTime="2.065230248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:53:41.552844921 +0000 UTC m=+340.127711210" watchObservedRunningTime="2026-09-07 03:53:41.553012062 +0000 UTC m=+340.127878351" Sep 07 03:53:41.937130 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:53:41.937040 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7827b2d1-c6cb-4b62-afff-6f22ff75e258" path="/var/lib/kubelet/pods/7827b2d1-c6cb-4b62-afff-6f22ff75e258/volumes" Sep 07 03:54:12.545320 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:12.545290 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-44pf7" Sep 07 03:54:45.227113 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.227078 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4"] Sep 07 03:54:45.227540 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.227389 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7827b2d1-c6cb-4b62-afff-6f22ff75e258" containerName="discovery" Sep 07 03:54:45.227540 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.227401 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7827b2d1-c6cb-4b62-afff-6f22ff75e258" containerName="discovery" Sep 07 03:54:45.227540 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.227456 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7827b2d1-c6cb-4b62-afff-6f22ff75e258" containerName="discovery" Sep 07 03:54:45.231664 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.231646 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:54:45.234736 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.234704 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 07 03:54:45.234839 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.234805 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 07 03:54:45.234839 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.234831 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lst6j\"" Sep 07 03:54:45.243240 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.243222 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4"] Sep 07 03:54:45.400134 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.400102 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88h9\" (UniqueName: \"kubernetes.io/projected/ce2ad692-c728-4484-954f-773faa78f5b4-kube-api-access-b88h9\") pod \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" (UID: \"ce2ad692-c728-4484-954f-773faa78f5b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:54:45.400294 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.400167 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce2ad692-c728-4484-954f-773faa78f5b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" (UID: \"ce2ad692-c728-4484-954f-773faa78f5b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:54:45.501036 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.500969 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce2ad692-c728-4484-954f-773faa78f5b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" (UID: \"ce2ad692-c728-4484-954f-773faa78f5b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:54:45.501036 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.501019 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88h9\" (UniqueName: \"kubernetes.io/projected/ce2ad692-c728-4484-954f-773faa78f5b4-kube-api-access-b88h9\") pod \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" (UID: \"ce2ad692-c728-4484-954f-773faa78f5b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:54:45.501330 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.501310 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce2ad692-c728-4484-954f-773faa78f5b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" (UID: \"ce2ad692-c728-4484-954f-773faa78f5b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:54:45.511477 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.511455 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88h9\" (UniqueName: \"kubernetes.io/projected/ce2ad692-c728-4484-954f-773faa78f5b4-kube-api-access-b88h9\") pod \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" (UID: \"ce2ad692-c728-4484-954f-773faa78f5b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:54:45.540982 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.540962 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:54:45.821214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:45.821139 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4"] Sep 07 03:54:45.824423 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:54:45.824392 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2ad692_c728_4484_954f_773faa78f5b4.slice/crio-f0d877394826a9db22a29aea6055adb8b6a856c5de1658a5df13eba7a1919b84 WatchSource:0}: Error finding container f0d877394826a9db22a29aea6055adb8b6a856c5de1658a5df13eba7a1919b84: Status 404 returned error can't find the container with id f0d877394826a9db22a29aea6055adb8b6a856c5de1658a5df13eba7a1919b84 Sep 07 03:54:46.815864 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:46.815830 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" event={"ID":"ce2ad692-c728-4484-954f-773faa78f5b4","Type":"ContainerStarted","Data":"f0d877394826a9db22a29aea6055adb8b6a856c5de1658a5df13eba7a1919b84"} Sep 07 03:54:47.970290 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:47.970254 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-xcn2k"] Sep 07 03:54:47.987134 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:47.986520 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-xcn2k" Sep 07 03:54:47.989123 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:47.989098 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-xcn2k"] Sep 07 03:54:47.991151 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:47.991116 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qf5nb\"" Sep 07 03:54:48.018271 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:48.018233 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hngkb\" (UniqueName: \"kubernetes.io/projected/c6a5a2bc-b46d-45ac-9dae-559d9fa2de80-kube-api-access-hngkb\") pod \"authorino-operator-68d88948d8-xcn2k\" (UID: \"c6a5a2bc-b46d-45ac-9dae-559d9fa2de80\") " pod="kuadrant-system/authorino-operator-68d88948d8-xcn2k" Sep 07 03:54:48.119514 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:48.119483 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hngkb\" (UniqueName: \"kubernetes.io/projected/c6a5a2bc-b46d-45ac-9dae-559d9fa2de80-kube-api-access-hngkb\") pod \"authorino-operator-68d88948d8-xcn2k\" (UID: \"c6a5a2bc-b46d-45ac-9dae-559d9fa2de80\") " pod="kuadrant-system/authorino-operator-68d88948d8-xcn2k" Sep 07 03:54:48.130909 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:48.130874 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hngkb\" (UniqueName: \"kubernetes.io/projected/c6a5a2bc-b46d-45ac-9dae-559d9fa2de80-kube-api-access-hngkb\") pod \"authorino-operator-68d88948d8-xcn2k\" (UID: \"c6a5a2bc-b46d-45ac-9dae-559d9fa2de80\") " pod="kuadrant-system/authorino-operator-68d88948d8-xcn2k" Sep 07 03:54:48.301412 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:48.301336 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-xcn2k" Sep 07 03:54:48.573474 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:48.573368 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-xcn2k"] Sep 07 03:54:48.576202 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:54:48.576174 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a5a2bc_b46d_45ac_9dae_559d9fa2de80.slice/crio-feade01832f4d2f92e54d443d1c8fba85c921d48ec0e028aede0bf7a3f7b6b20 WatchSource:0}: Error finding container feade01832f4d2f92e54d443d1c8fba85c921d48ec0e028aede0bf7a3f7b6b20: Status 404 returned error can't find the container with id feade01832f4d2f92e54d443d1c8fba85c921d48ec0e028aede0bf7a3f7b6b20 Sep 07 03:54:48.826786 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:48.826712 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-xcn2k" event={"ID":"c6a5a2bc-b46d-45ac-9dae-559d9fa2de80","Type":"ContainerStarted","Data":"feade01832f4d2f92e54d443d1c8fba85c921d48ec0e028aede0bf7a3f7b6b20"} Sep 07 03:54:50.046591 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:50.046558 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt"] Sep 07 03:54:50.049945 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:50.049926 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:54:50.052772 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:50.052749 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8fqqg\"" Sep 07 03:54:50.057848 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:50.057826 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt"] Sep 07 03:54:50.131777 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:50.131753 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8f88\" (UniqueName: \"kubernetes.io/projected/83e6bfb8-46c4-4549-82be-bcd4eaf2350b-kube-api-access-k8f88\") pod \"limitador-operator-controller-manager-6c549b6d85-b67xt\" (UID: \"83e6bfb8-46c4-4549-82be-bcd4eaf2350b\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:54:50.232614 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:50.232580 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8f88\" (UniqueName: \"kubernetes.io/projected/83e6bfb8-46c4-4549-82be-bcd4eaf2350b-kube-api-access-k8f88\") pod \"limitador-operator-controller-manager-6c549b6d85-b67xt\" (UID: \"83e6bfb8-46c4-4549-82be-bcd4eaf2350b\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:54:50.240990 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:50.240966 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8f88\" (UniqueName: \"kubernetes.io/projected/83e6bfb8-46c4-4549-82be-bcd4eaf2350b-kube-api-access-k8f88\") pod \"limitador-operator-controller-manager-6c549b6d85-b67xt\" (UID: \"83e6bfb8-46c4-4549-82be-bcd4eaf2350b\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:54:50.363593 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:50.363529 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:54:50.646676 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:50.646595 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt"] Sep 07 03:54:50.887808 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:54:50.887767 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e6bfb8_46c4_4549_82be_bcd4eaf2350b.slice/crio-ed2db90a8d489b94f3bd745ffde1a988d99cb39e1f2a56c4cd4df29ed1df4eec WatchSource:0}: Error finding container ed2db90a8d489b94f3bd745ffde1a988d99cb39e1f2a56c4cd4df29ed1df4eec: Status 404 returned error can't find the container with id ed2db90a8d489b94f3bd745ffde1a988d99cb39e1f2a56c4cd4df29ed1df4eec Sep 07 03:54:51.844248 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:51.844209 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-xcn2k" event={"ID":"c6a5a2bc-b46d-45ac-9dae-559d9fa2de80","Type":"ContainerStarted","Data":"491e60228a42b6c5d3c19be610ce78234b9d1440ceedaef0bc97748627df38e3"} Sep 07 03:54:51.844701 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:51.844313 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-xcn2k" Sep 07 03:54:51.845727 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:51.845703 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" event={"ID":"83e6bfb8-46c4-4549-82be-bcd4eaf2350b","Type":"ContainerStarted","Data":"ed2db90a8d489b94f3bd745ffde1a988d99cb39e1f2a56c4cd4df29ed1df4eec"} Sep 07 03:54:51.862597 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:51.862546 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-xcn2k" podStartSLOduration=2.498354729 podStartE2EDuration="4.862528173s" totalImagesPullingTime="2.364173444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:54:51.860984111 +0000 UTC m=+410.435850425" watchObservedRunningTime="2026-09-07 03:54:51.862528173 +0000 UTC m=+410.437394463" Sep 07 03:54:52.852217 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:52.852128 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" event={"ID":"ce2ad692-c728-4484-954f-773faa78f5b4","Type":"ContainerStarted","Data":"fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3"} Sep 07 03:54:52.852654 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:52.852221 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:54:52.883498 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:52.883451 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" podStartSLOduration=1.237489176 podStartE2EDuration="7.883436809s" totalImagesPullingTime="6.645947633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:54:52.880822897 +0000 UTC m=+411.455689185" watchObservedRunningTime="2026-09-07 03:54:52.883436809 +0000 UTC m=+411.458303097" Sep 07 03:54:54.863586 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:54.863552 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" event={"ID":"83e6bfb8-46c4-4549-82be-bcd4eaf2350b","Type":"ContainerStarted","Data":"4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4"} Sep 07 03:54:54.863951 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:54.863629 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:54:54.879602 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:54:54.879562 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" podStartSLOduration=1.490225271 podStartE2EDuration="4.87954688s" totalImagesPullingTime="3.389321609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:54:54.878523284 +0000 UTC m=+413.453389576" watchObservedRunningTime="2026-09-07 03:54:54.87954688 +0000 UTC m=+413.454413167" Sep 07 03:55:02.854300 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:02.854270 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-xcn2k" Sep 07 03:55:03.859846 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:03.859818 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:55:05.843377 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.843342 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4"] Sep 07 03:55:05.846844 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.846821 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:05.865190 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.865163 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4"] Sep 07 03:55:05.870367 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.870344 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:55:05.928003 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.927969 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4"] Sep 07 03:55:05.928176 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.928154 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" containerName="manager" containerID="cri-o://fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3" gracePeriod=2 Sep 07 03:55:05.939151 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.938970 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4"] Sep 07 03:55:05.950142 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.950119 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4"] Sep 07 03:55:05.950398 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:05.950378 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-fszjx], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" podUID="fc55e896-44d3-40d5-9d06-f5d44e93414e" Sep 07 03:55:05.955477 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.955447 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f"] Sep 07 03:55:05.955959 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.955942 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" containerName="manager" Sep 07 03:55:05.956001 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.955963 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" containerName="manager" Sep 07 03:55:05.956092 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.956050 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" containerName="manager" Sep 07 03:55:05.959246 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.959227 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4"] Sep 07 03:55:05.959326 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.959310 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:05.968459 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.968424 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt"] Sep 07 03:55:05.968602 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.968582 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" containerName="manager" containerID="cri-o://4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4" gracePeriod=2 Sep 07 03:55:05.968989 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.968955 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fszjx\" (UniqueName: \"kubernetes.io/projected/fc55e896-44d3-40d5-9d06-f5d44e93414e-kube-api-access-fszjx\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gjwq4\" (UID: \"fc55e896-44d3-40d5-9d06-f5d44e93414e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:05.969120 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.968991 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc55e896-44d3-40d5-9d06-f5d44e93414e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gjwq4\" (UID: \"fc55e896-44d3-40d5-9d06-f5d44e93414e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:05.971650 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.971628 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f"] Sep 07 03:55:05.980275 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.980252 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk"] Sep 07 03:55:05.983875 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.983854 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt"] Sep 07 03:55:05.983978 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.983944 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 03:55:05.999166 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:05.999140 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk"] Sep 07 03:55:06.001796 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.001773 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2"] Sep 07 03:55:06.002232 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.002213 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" containerName="manager" Sep 07 03:55:06.002232 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.002232 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" containerName="manager" Sep 07 03:55:06.002396 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.002326 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" containerName="manager" Sep 07 03:55:06.005232 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.005214 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2" Sep 07 03:55:06.009935 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.009902 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:55:06.024884 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.024858 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2"] Sep 07 03:55:06.043283 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.043243 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:55:06.045202 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.045175 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-b67xt\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:55:06.070201 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.070174 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrljk\" (UniqueName: \"kubernetes.io/projected/91846c5b-9098-450e-82ba-e782fe2b1c74-kube-api-access-nrljk\") pod \"limitador-operator-controller-manager-6c549b6d85-g28g2\" (UID: \"91846c5b-9098-450e-82ba-e782fe2b1c74\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2" Sep 07 03:55:06.070288 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.070222 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e854fd0d-cc2b-42d5-80f9-a9615037b862-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-qdw6f\" (UID: \"e854fd0d-cc2b-42d5-80f9-a9615037b862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:06.070288 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.070273 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2k72\" (UniqueName: \"kubernetes.io/projected/176fc359-0dc1-4fbe-854e-9e7daccb381b-kube-api-access-s2k72\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-7vrpk\" (UID: \"176fc359-0dc1-4fbe-854e-9e7daccb381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 03:55:06.070370 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.070295 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd5bp\" (UniqueName: \"kubernetes.io/projected/e854fd0d-cc2b-42d5-80f9-a9615037b862-kube-api-access-gd5bp\") pod \"kuadrant-operator-controller-manager-58fc56757c-qdw6f\" (UID: \"e854fd0d-cc2b-42d5-80f9-a9615037b862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:06.070370 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.070344 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176fc359-0dc1-4fbe-854e-9e7daccb381b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-7vrpk\" (UID: \"176fc359-0dc1-4fbe-854e-9e7daccb381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 03:55:06.070461 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.070368 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszjx\" (UniqueName: \"kubernetes.io/projected/fc55e896-44d3-40d5-9d06-f5d44e93414e-kube-api-access-fszjx\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gjwq4\" (UID: \"fc55e896-44d3-40d5-9d06-f5d44e93414e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:06.070461 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.070391 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc55e896-44d3-40d5-9d06-f5d44e93414e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gjwq4\" (UID: \"fc55e896-44d3-40d5-9d06-f5d44e93414e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:06.070750 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.070733 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc55e896-44d3-40d5-9d06-f5d44e93414e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gjwq4\" (UID: \"fc55e896-44d3-40d5-9d06-f5d44e93414e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:06.075491 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.075475 2555 projected.go:196] Error preparing data for projected volume kube-api-access-fszjx for pod kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4: failed to fetch token: pod "kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" not found Sep 07 03:55:06.075630 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.075536 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc55e896-44d3-40d5-9d06-f5d44e93414e-kube-api-access-fszjx podName:fc55e896-44d3-40d5-9d06-f5d44e93414e nodeName:}" failed. No retries permitted until 2026-09-07 03:55:06.575521408 +0000 UTC m=+425.150387676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fszjx" (UniqueName: "kubernetes.io/projected/fc55e896-44d3-40d5-9d06-f5d44e93414e-kube-api-access-fszjx") pod "kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" (UID: "fc55e896-44d3-40d5-9d06-f5d44e93414e") : failed to fetch token: pod "kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" not found Sep 07 03:55:06.171557 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.171524 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrljk\" (UniqueName: \"kubernetes.io/projected/91846c5b-9098-450e-82ba-e782fe2b1c74-kube-api-access-nrljk\") pod \"limitador-operator-controller-manager-6c549b6d85-g28g2\" (UID: \"91846c5b-9098-450e-82ba-e782fe2b1c74\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2" Sep 07 03:55:06.171721 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.171573 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e854fd0d-cc2b-42d5-80f9-a9615037b862-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-qdw6f\" (UID: \"e854fd0d-cc2b-42d5-80f9-a9615037b862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:06.171721 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.171702 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2k72\" (UniqueName: \"kubernetes.io/projected/176fc359-0dc1-4fbe-854e-9e7daccb381b-kube-api-access-s2k72\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-7vrpk\" (UID: \"176fc359-0dc1-4fbe-854e-9e7daccb381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 03:55:06.171824 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.171729 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd5bp\" (UniqueName: \"kubernetes.io/projected/e854fd0d-cc2b-42d5-80f9-a9615037b862-kube-api-access-gd5bp\") pod \"kuadrant-operator-controller-manager-58fc56757c-qdw6f\" (UID: \"e854fd0d-cc2b-42d5-80f9-a9615037b862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:06.171824 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.171785 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176fc359-0dc1-4fbe-854e-9e7daccb381b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-7vrpk\" (UID: \"176fc359-0dc1-4fbe-854e-9e7daccb381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 03:55:06.172200 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.172176 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176fc359-0dc1-4fbe-854e-9e7daccb381b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-7vrpk\" (UID: \"176fc359-0dc1-4fbe-854e-9e7daccb381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 03:55:06.172297 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.172176 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e854fd0d-cc2b-42d5-80f9-a9615037b862-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-qdw6f\" (UID: \"e854fd0d-cc2b-42d5-80f9-a9615037b862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:06.180381 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.180359 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2k72\" (UniqueName: \"kubernetes.io/projected/176fc359-0dc1-4fbe-854e-9e7daccb381b-kube-api-access-s2k72\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-7vrpk\" (UID: \"176fc359-0dc1-4fbe-854e-9e7daccb381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 03:55:06.180527 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.180503 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrljk\" (UniqueName: \"kubernetes.io/projected/91846c5b-9098-450e-82ba-e782fe2b1c74-kube-api-access-nrljk\") pod \"limitador-operator-controller-manager-6c549b6d85-g28g2\" (UID: \"91846c5b-9098-450e-82ba-e782fe2b1c74\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2" Sep 07 03:55:06.180566 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.180509 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd5bp\" (UniqueName: \"kubernetes.io/projected/e854fd0d-cc2b-42d5-80f9-a9615037b862-kube-api-access-gd5bp\") pod \"kuadrant-operator-controller-manager-58fc56757c-qdw6f\" (UID: \"e854fd0d-cc2b-42d5-80f9-a9615037b862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:06.211859 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.211834 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:55:06.213864 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.213848 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:55:06.214118 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.214098 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:55:06.216077 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.216041 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-b67xt\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:55:06.217847 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.217829 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:55:06.221257 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.221238 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-b67xt\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:55:06.272274 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.272247 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b88h9\" (UniqueName: \"kubernetes.io/projected/ce2ad692-c728-4484-954f-773faa78f5b4-kube-api-access-b88h9\") pod \"ce2ad692-c728-4484-954f-773faa78f5b4\" (UID: \"ce2ad692-c728-4484-954f-773faa78f5b4\") " Sep 07 03:55:06.272381 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.272282 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce2ad692-c728-4484-954f-773faa78f5b4-extensions-socket-volume\") pod \"ce2ad692-c728-4484-954f-773faa78f5b4\" (UID: \"ce2ad692-c728-4484-954f-773faa78f5b4\") " Sep 07 03:55:06.272381 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.272357 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8f88\" (UniqueName: \"kubernetes.io/projected/83e6bfb8-46c4-4549-82be-bcd4eaf2350b-kube-api-access-k8f88\") pod \"83e6bfb8-46c4-4549-82be-bcd4eaf2350b\" (UID: \"83e6bfb8-46c4-4549-82be-bcd4eaf2350b\") " Sep 07 03:55:06.272705 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.272680 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2ad692-c728-4484-954f-773faa78f5b4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ce2ad692-c728-4484-954f-773faa78f5b4" (UID: "ce2ad692-c728-4484-954f-773faa78f5b4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 03:55:06.274422 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.274397 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2ad692-c728-4484-954f-773faa78f5b4-kube-api-access-b88h9" (OuterVolumeSpecName: "kube-api-access-b88h9") pod "ce2ad692-c728-4484-954f-773faa78f5b4" (UID: "ce2ad692-c728-4484-954f-773faa78f5b4"). InnerVolumeSpecName "kube-api-access-b88h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:55:06.274484 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.274435 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e6bfb8-46c4-4549-82be-bcd4eaf2350b-kube-api-access-k8f88" (OuterVolumeSpecName: "kube-api-access-k8f88") pod "83e6bfb8-46c4-4549-82be-bcd4eaf2350b" (UID: "83e6bfb8-46c4-4549-82be-bcd4eaf2350b"). InnerVolumeSpecName "kube-api-access-k8f88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:55:06.353114 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.353079 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:06.360042 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.360016 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 03:55:06.366947 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.366922 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2" Sep 07 03:55:06.373383 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.373352 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88h9\" (UniqueName: \"kubernetes.io/projected/ce2ad692-c728-4484-954f-773faa78f5b4-kube-api-access-b88h9\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:55:06.373383 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.373382 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce2ad692-c728-4484-954f-773faa78f5b4-extensions-socket-volume\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:55:06.373538 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.373398 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8f88\" (UniqueName: \"kubernetes.io/projected/83e6bfb8-46c4-4549-82be-bcd4eaf2350b-kube-api-access-k8f88\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:55:06.638359 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.638335 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f"] Sep 07 03:55:06.641517 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:55:06.641490 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode854fd0d_cc2b_42d5_80f9_a9615037b862.slice/crio-c3db893f63bce6bcc4aa7f6f4786a5b9f69f4b0ca38199bbac0d25017296db3a WatchSource:0}: Error finding container c3db893f63bce6bcc4aa7f6f4786a5b9f69f4b0ca38199bbac0d25017296db3a: Status 404 returned error can't find the container with id c3db893f63bce6bcc4aa7f6f4786a5b9f69f4b0ca38199bbac0d25017296db3a Sep 07 03:55:06.642047 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.642022 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk"] Sep 07 03:55:06.643754 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:55:06.643730 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176fc359_0dc1_4fbe_854e_9e7daccb381b.slice/crio-8f8c8c99e39d875cf89a05995a705acdb8a22ae0d8a431588d24effadb5dbdc5 WatchSource:0}: Error finding container 8f8c8c99e39d875cf89a05995a705acdb8a22ae0d8a431588d24effadb5dbdc5: Status 404 returned error can't find the container with id 8f8c8c99e39d875cf89a05995a705acdb8a22ae0d8a431588d24effadb5dbdc5 Sep 07 03:55:06.655578 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.655525 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2"] Sep 07 03:55:06.662048 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:55:06.662022 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91846c5b_9098_450e_82ba_e782fe2b1c74.slice/crio-812edd0771e68cd478cae7b27522bb953b58c3fa22660635ce512936721fa568 WatchSource:0}: Error finding container 812edd0771e68cd478cae7b27522bb953b58c3fa22660635ce512936721fa568: Status 404 returned error can't find the container with id 812edd0771e68cd478cae7b27522bb953b58c3fa22660635ce512936721fa568 Sep 07 03:55:06.676012 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.675985 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszjx\" (UniqueName: \"kubernetes.io/projected/fc55e896-44d3-40d5-9d06-f5d44e93414e-kube-api-access-fszjx\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gjwq4\" (UID: \"fc55e896-44d3-40d5-9d06-f5d44e93414e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:06.679239 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.679219 2555 projected.go:196] Error preparing data for projected volume kube-api-access-fszjx for pod kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4: failed to fetch token: pod "kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" not found Sep 07 03:55:06.679338 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.679275 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc55e896-44d3-40d5-9d06-f5d44e93414e-kube-api-access-fszjx podName:fc55e896-44d3-40d5-9d06-f5d44e93414e nodeName:}" failed. No retries permitted until 2026-09-07 03:55:07.679260594 +0000 UTC m=+426.254126866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fszjx" (UniqueName: "kubernetes.io/projected/fc55e896-44d3-40d5-9d06-f5d44e93414e-kube-api-access-fszjx") pod "kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" (UID: "fc55e896-44d3-40d5-9d06-f5d44e93414e") : failed to fetch token: pod "kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" not found Sep 07 03:55:06.919393 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.919311 2555 generic.go:356] "Generic (PLEG): container finished" podID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" containerID="4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4" exitCode=0 Sep 07 03:55:06.919393 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.919353 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:55:06.919833 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.919409 2555 scope.go:117] "RemoveContainer" containerID="4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4" Sep 07 03:55:06.921451 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.921419 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" event={"ID":"e854fd0d-cc2b-42d5-80f9-a9615037b862","Type":"ContainerStarted","Data":"93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec"} Sep 07 03:55:06.921696 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.921479 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" event={"ID":"e854fd0d-cc2b-42d5-80f9-a9615037b862","Type":"ContainerStarted","Data":"c3db893f63bce6bcc4aa7f6f4786a5b9f69f4b0ca38199bbac0d25017296db3a"} Sep 07 03:55:06.921696 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.921545 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:06.921840 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.921787 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:55:06.922931 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.922912 2555 generic.go:356] "Generic (PLEG): container finished" podID="ce2ad692-c728-4484-954f-773faa78f5b4" containerID="fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3" exitCode=0 Sep 07 03:55:06.923029 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.922944 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:55:06.923809 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.923776 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-b67xt\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:55:06.929867 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.926013 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:55:06.929867 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.928209 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-b67xt\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:55:06.932630 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.932467 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2" event={"ID":"91846c5b-9098-450e-82ba-e782fe2b1c74","Type":"ContainerStarted","Data":"8110d2696289c5c5dec0b2190ee8bddb8cb3419fc88725e66934daf3ada04f28"} Sep 07 03:55:06.932630 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.932566 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2" event={"ID":"91846c5b-9098-450e-82ba-e782fe2b1c74","Type":"ContainerStarted","Data":"812edd0771e68cd478cae7b27522bb953b58c3fa22660635ce512936721fa568"} Sep 07 03:55:06.933390 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.932716 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2" Sep 07 03:55:06.936174 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.936020 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:06.936174 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.936130 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" event={"ID":"176fc359-0dc1-4fbe-854e-9e7daccb381b","Type":"ContainerStarted","Data":"986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd"} Sep 07 03:55:06.936315 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.936174 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" event={"ID":"176fc359-0dc1-4fbe-854e-9e7daccb381b","Type":"ContainerStarted","Data":"8f8c8c99e39d875cf89a05995a705acdb8a22ae0d8a431588d24effadb5dbdc5"} Sep 07 03:55:06.937780 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.937486 2555 scope.go:117] "RemoveContainer" containerID="4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4" Sep 07 03:55:06.937896 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.937857 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4\": container with ID starting with 4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4 not found: ID does not exist" containerID="4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4" Sep 07 03:55:06.937975 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.937891 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4"} err="failed to get container status \"4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4\": rpc error: code = NotFound desc = could not find container \"4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4\": container with ID starting with 4a11fdee39d97d14ee55a0f7b444284bf680a38013a558e7bc7ae0b95cc5f6c4 not found: ID does not exist" Sep 07 03:55:06.937975 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.937913 2555 scope.go:117] "RemoveContainer" containerID="fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3" Sep 07 03:55:06.946444 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.946423 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:06.949808 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.949769 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" podStartSLOduration=1.949757661 podStartE2EDuration="1.949757661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:55:06.949538676 +0000 UTC m=+425.524404966" watchObservedRunningTime="2026-09-07 03:55:06.949757661 +0000 UTC m=+425.524623950" Sep 07 03:55:06.952145 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.952123 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5b65f94d4c-gjwq4\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="fc55e896-44d3-40d5-9d06-f5d44e93414e" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:06.963161 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.963127 2555 scope.go:117] "RemoveContainer" containerID="fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3" Sep 07 03:55:06.963438 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.963421 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3\": container with ID starting with fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3 not found: ID does not exist" containerID="fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3" Sep 07 03:55:06.963492 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.963450 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3"} err="failed to get container status \"fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3\": rpc error: code = NotFound desc = could not find container \"fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3\": container with ID starting with fab9add636e67d486a58ed88e94a5581c0727321f7dd19e2b94df7bc9e50b9f3 not found: ID does not exist" Sep 07 03:55:06.975810 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:06.975762 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" podStartSLOduration=1.9757469680000002 podStartE2EDuration="1.975746968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:55:06.974870154 +0000 UTC m=+425.549736443" watchObservedRunningTime="2026-09-07 03:55:06.975746968 +0000 UTC m=+425.550613258" Sep 07 03:55:06.979787 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.979764 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vntn4\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vntn4" Sep 07 03:55:06.981881 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:06.981862 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-b67xt\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b67xt" Sep 07 03:55:07.001279 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:07.001238 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2" podStartSLOduration=2.001226908 podStartE2EDuration="2.001226908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:55:06.999365943 +0000 UTC m=+425.574232244" watchObservedRunningTime="2026-09-07 03:55:07.001226908 +0000 UTC m=+425.576093208" Sep 07 03:55:07.079420 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:07.079388 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc55e896-44d3-40d5-9d06-f5d44e93414e-extensions-socket-volume\") pod \"fc55e896-44d3-40d5-9d06-f5d44e93414e\" (UID: \"fc55e896-44d3-40d5-9d06-f5d44e93414e\") " Sep 07 03:55:07.079613 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:07.079600 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fszjx\" (UniqueName: \"kubernetes.io/projected/fc55e896-44d3-40d5-9d06-f5d44e93414e-kube-api-access-fszjx\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:55:07.079662 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:07.079615 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc55e896-44d3-40d5-9d06-f5d44e93414e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fc55e896-44d3-40d5-9d06-f5d44e93414e" (UID: "fc55e896-44d3-40d5-9d06-f5d44e93414e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 03:55:07.180776 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:07.180692 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc55e896-44d3-40d5-9d06-f5d44e93414e-extensions-socket-volume\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:55:07.936832 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:07.936798 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e6bfb8-46c4-4549-82be-bcd4eaf2350b" path="/var/lib/kubelet/pods/83e6bfb8-46c4-4549-82be-bcd4eaf2350b/volumes" Sep 07 03:55:07.937245 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:07.937231 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2ad692-c728-4484-954f-773faa78f5b4" path="/var/lib/kubelet/pods/ce2ad692-c728-4484-954f-773faa78f5b4/volumes" Sep 07 03:55:07.937523 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:07.937510 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc55e896-44d3-40d5-9d06-f5d44e93414e" path="/var/lib/kubelet/pods/fc55e896-44d3-40d5-9d06-f5d44e93414e/volumes" Sep 07 03:55:07.942345 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:07.942314 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:07.942847 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:07.942825 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 03:55:07.948037 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:07.948015 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5b65f94d4c-gjwq4\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="fc55e896-44d3-40d5-9d06-f5d44e93414e" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gjwq4" Sep 07 03:55:17.944962 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:17.944934 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-g28g2" Sep 07 03:55:17.945505 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:17.944998 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:18.949607 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:18.949581 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 03:55:19.003162 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:19.003130 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f"] Sep 07 03:55:19.003352 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:19.003327 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" podUID="e854fd0d-cc2b-42d5-80f9-a9615037b862" containerName="manager" containerID="cri-o://93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec" gracePeriod=10 Sep 07 03:55:19.246099 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:19.246076 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:19.383865 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:19.383840 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e854fd0d-cc2b-42d5-80f9-a9615037b862-extensions-socket-volume\") pod \"e854fd0d-cc2b-42d5-80f9-a9615037b862\" (UID: \"e854fd0d-cc2b-42d5-80f9-a9615037b862\") " Sep 07 03:55:19.384019 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:19.383937 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd5bp\" (UniqueName: \"kubernetes.io/projected/e854fd0d-cc2b-42d5-80f9-a9615037b862-kube-api-access-gd5bp\") pod \"e854fd0d-cc2b-42d5-80f9-a9615037b862\" (UID: \"e854fd0d-cc2b-42d5-80f9-a9615037b862\") " Sep 07 03:55:19.384277 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:19.384252 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e854fd0d-cc2b-42d5-80f9-a9615037b862-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e854fd0d-cc2b-42d5-80f9-a9615037b862" (UID: "e854fd0d-cc2b-42d5-80f9-a9615037b862"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 03:55:19.385834 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:19.385809 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e854fd0d-cc2b-42d5-80f9-a9615037b862-kube-api-access-gd5bp" (OuterVolumeSpecName: "kube-api-access-gd5bp") pod "e854fd0d-cc2b-42d5-80f9-a9615037b862" (UID: "e854fd0d-cc2b-42d5-80f9-a9615037b862"). InnerVolumeSpecName "kube-api-access-gd5bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:55:19.485389 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:19.485319 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd5bp\" (UniqueName: \"kubernetes.io/projected/e854fd0d-cc2b-42d5-80f9-a9615037b862-kube-api-access-gd5bp\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:55:19.485389 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:19.485355 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e854fd0d-cc2b-42d5-80f9-a9615037b862-extensions-socket-volume\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:55:20.005344 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:20.005310 2555 generic.go:356] "Generic (PLEG): container finished" podID="e854fd0d-cc2b-42d5-80f9-a9615037b862" containerID="93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec" exitCode=0 Sep 07 03:55:20.005734 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:20.005358 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" Sep 07 03:55:20.005734 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:20.005392 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" event={"ID":"e854fd0d-cc2b-42d5-80f9-a9615037b862","Type":"ContainerDied","Data":"93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec"} Sep 07 03:55:20.005734 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:20.005446 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f" event={"ID":"e854fd0d-cc2b-42d5-80f9-a9615037b862","Type":"ContainerDied","Data":"c3db893f63bce6bcc4aa7f6f4786a5b9f69f4b0ca38199bbac0d25017296db3a"} Sep 07 03:55:20.005734 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:20.005468 2555 scope.go:117] "RemoveContainer" containerID="93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec" Sep 07 03:55:20.013724 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:20.013707 2555 scope.go:117] "RemoveContainer" containerID="93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec" Sep 07 03:55:20.014015 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:20.013997 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec\": container with ID starting with 93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec not found: ID does not exist" containerID="93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec" Sep 07 03:55:20.014078 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:20.014027 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec"} err="failed to get container status \"93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec\": rpc error: code = NotFound desc = could not find container \"93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec\": container with ID starting with 93584ca9198b8f7039cd413c048e7db42a3c16acc954871525ea7e725006e3ec not found: ID does not exist" Sep 07 03:55:20.025164 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:20.025143 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f"] Sep 07 03:55:20.033470 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:20.033452 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qdw6f"] Sep 07 03:55:21.936233 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:21.936201 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e854fd0d-cc2b-42d5-80f9-a9615037b862" path="/var/lib/kubelet/pods/e854fd0d-cc2b-42d5-80f9-a9615037b862/volumes" Sep 07 03:55:35.253416 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.253380 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb"] Sep 07 03:55:35.253886 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.253722 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e854fd0d-cc2b-42d5-80f9-a9615037b862" containerName="manager" Sep 07 03:55:35.253886 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.253734 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e854fd0d-cc2b-42d5-80f9-a9615037b862" containerName="manager" Sep 07 03:55:35.253886 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.253809 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e854fd0d-cc2b-42d5-80f9-a9615037b862" containerName="manager" Sep 07 03:55:35.256800 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.256777 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.259367 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.259346 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kcv6s\"" Sep 07 03:55:35.259482 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.259403 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 07 03:55:35.269199 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.269176 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb"] Sep 07 03:55:35.407608 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.407571 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.407608 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.407614 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9bed40-0504-4031-b120-d3d199e315a6-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.407869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.407687 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9bed40-0504-4031-b120-d3d199e315a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.407869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.407730 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.407869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.407794 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.407869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.407826 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9bed40-0504-4031-b120-d3d199e315a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.407869 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.407855 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.408109 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.407888 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.408109 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.407919 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr6sv\" (UniqueName: \"kubernetes.io/projected/ab9bed40-0504-4031-b120-d3d199e315a6-kube-api-access-qr6sv\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509129 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509166 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9bed40-0504-4031-b120-d3d199e315a6-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509196 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9bed40-0504-4031-b120-d3d199e315a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509214 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509217 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509539 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509248 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509539 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509269 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9bed40-0504-4031-b120-d3d199e315a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509539 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509289 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509539 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509319 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509539 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509347 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr6sv\" (UniqueName: \"kubernetes.io/projected/ab9bed40-0504-4031-b120-d3d199e315a6-kube-api-access-qr6sv\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509857 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509648 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509857 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509733 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.509971 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.509854 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.510132 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.510113 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.510205 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.510129 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9bed40-0504-4031-b120-d3d199e315a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.511540 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.511521 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9bed40-0504-4031-b120-d3d199e315a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.511726 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.511710 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9bed40-0504-4031-b120-d3d199e315a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.517215 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.517193 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr6sv\" (UniqueName: \"kubernetes.io/projected/ab9bed40-0504-4031-b120-d3d199e315a6-kube-api-access-qr6sv\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.517328 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.517223 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9bed40-0504-4031-b120-d3d199e315a6-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-d7pgb\" (UID: \"ab9bed40-0504-4031-b120-d3d199e315a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.567348 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.567325 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:35.844376 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:35.844351 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb"] Sep 07 03:55:35.846335 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:55:35.846294 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9bed40_0504_4031_b120_d3d199e315a6.slice/crio-ee34fa0ce4164b7b514917556467d32ab6f1e4e3ace08f1f9e3c75c18279838a WatchSource:0}: Error finding container ee34fa0ce4164b7b514917556467d32ab6f1e4e3ace08f1f9e3c75c18279838a: Status 404 returned error can't find the container with id ee34fa0ce4164b7b514917556467d32ab6f1e4e3ace08f1f9e3c75c18279838a Sep 07 03:55:36.088873 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:36.088836 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" event={"ID":"ab9bed40-0504-4031-b120-d3d199e315a6","Type":"ContainerStarted","Data":"ee34fa0ce4164b7b514917556467d32ab6f1e4e3ace08f1f9e3c75c18279838a"} Sep 07 03:55:38.558653 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:38.558616 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 03:55:38.558943 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:38.558690 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 03:55:38.558943 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:38.558718 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 03:55:39.107447 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:39.107409 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" event={"ID":"ab9bed40-0504-4031-b120-d3d199e315a6","Type":"ContainerStarted","Data":"77881066059cf2313bb0980e8a3b3524bea6a4693c3468d356202d68c451ccdc"} Sep 07 03:55:39.132672 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:39.132626 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" podStartSLOduration=1.422681728 podStartE2EDuration="4.132610367s" totalImagesPullingTime="2.709928639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:55:39.131232045 +0000 UTC m=+457.706098345" watchObservedRunningTime="2026-09-07 03:55:39.132610367 +0000 UTC m=+457.707476655" Sep 07 03:55:39.568162 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:39.568119 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:39.572832 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:39.572808 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:40.112764 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.112732 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:40.113700 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.113676 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:55:40.340345 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.340314 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:55:40.343950 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.343921 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" Sep 07 03:55:40.346371 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.346344 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 07 03:55:40.346485 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.346421 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-l4lt8\"" Sep 07 03:55:40.353109 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.353047 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8-config-file\") pod \"limitador-limitador-85d9db4d5d-7tl54\" (UID: \"79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" Sep 07 03:55:40.353237 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.353136 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csbf7\" (UniqueName: \"kubernetes.io/projected/79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8-kube-api-access-csbf7\") pod \"limitador-limitador-85d9db4d5d-7tl54\" (UID: \"79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" Sep 07 03:55:40.354859 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.354826 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:55:40.388381 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.388296 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:55:40.454305 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.454270 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8-config-file\") pod \"limitador-limitador-85d9db4d5d-7tl54\" (UID: \"79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" Sep 07 03:55:40.454476 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.454312 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csbf7\" (UniqueName: \"kubernetes.io/projected/79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8-kube-api-access-csbf7\") pod \"limitador-limitador-85d9db4d5d-7tl54\" (UID: \"79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" Sep 07 03:55:40.455178 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.455157 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8-config-file\") pod \"limitador-limitador-85d9db4d5d-7tl54\" (UID: \"79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" Sep 07 03:55:40.462751 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.462710 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csbf7\" (UniqueName: \"kubernetes.io/projected/79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8-kube-api-access-csbf7\") pod \"limitador-limitador-85d9db4d5d-7tl54\" (UID: \"79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" Sep 07 03:55:40.655791 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.655703 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" Sep 07 03:55:40.927997 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:40.927971 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:55:40.930505 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:55:40.930476 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d04ba1_10bd_4364_ace4_ecd7ecc1a3f8.slice/crio-7858f37b806cd1c3150adcfe647950f1d54e3b8962a7c3e596b4d9bba095392d WatchSource:0}: Error finding container 7858f37b806cd1c3150adcfe647950f1d54e3b8962a7c3e596b4d9bba095392d: Status 404 returned error can't find the container with id 7858f37b806cd1c3150adcfe647950f1d54e3b8962a7c3e596b4d9bba095392d Sep 07 03:55:41.003443 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:41.003407 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8h7l"] Sep 07 03:55:41.008012 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:41.007987 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8h7l" Sep 07 03:55:41.010625 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:41.010596 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mws5h\"" Sep 07 03:55:41.015768 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:41.015605 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8h7l"] Sep 07 03:55:41.059704 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:41.059669 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9r6c\" (UniqueName: \"kubernetes.io/projected/c75f1de9-7bb2-4720-b121-bb7bb01998f0-kube-api-access-h9r6c\") pod \"authorino-b54df94b4-b8h7l\" (UID: \"c75f1de9-7bb2-4720-b121-bb7bb01998f0\") " pod="kuadrant-system/authorino-b54df94b4-b8h7l" Sep 07 03:55:41.118650 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:41.118616 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" event={"ID":"79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8","Type":"ContainerStarted","Data":"7858f37b806cd1c3150adcfe647950f1d54e3b8962a7c3e596b4d9bba095392d"} Sep 07 03:55:41.160777 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:41.160734 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9r6c\" (UniqueName: \"kubernetes.io/projected/c75f1de9-7bb2-4720-b121-bb7bb01998f0-kube-api-access-h9r6c\") pod \"authorino-b54df94b4-b8h7l\" (UID: \"c75f1de9-7bb2-4720-b121-bb7bb01998f0\") " pod="kuadrant-system/authorino-b54df94b4-b8h7l" Sep 07 03:55:41.171916 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:41.171889 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9r6c\" (UniqueName: \"kubernetes.io/projected/c75f1de9-7bb2-4720-b121-bb7bb01998f0-kube-api-access-h9r6c\") pod \"authorino-b54df94b4-b8h7l\" (UID: \"c75f1de9-7bb2-4720-b121-bb7bb01998f0\") " pod="kuadrant-system/authorino-b54df94b4-b8h7l" Sep 07 03:55:41.318982 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:41.318933 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8h7l" Sep 07 03:55:41.588774 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:41.588639 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8h7l"] Sep 07 03:55:41.591312 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:55:41.591286 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc75f1de9_7bb2_4720_b121_bb7bb01998f0.slice/crio-a4bdcd2b3bb00b50843dbf1ffd96e4c316e81c5d62e2cd2f2a099035ce7cd860 WatchSource:0}: Error finding container a4bdcd2b3bb00b50843dbf1ffd96e4c316e81c5d62e2cd2f2a099035ce7cd860: Status 404 returned error can't find the container with id a4bdcd2b3bb00b50843dbf1ffd96e4c316e81c5d62e2cd2f2a099035ce7cd860 Sep 07 03:55:42.125677 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:42.125642 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8h7l" event={"ID":"c75f1de9-7bb2-4720-b121-bb7bb01998f0","Type":"ContainerStarted","Data":"a4bdcd2b3bb00b50843dbf1ffd96e4c316e81c5d62e2cd2f2a099035ce7cd860"} Sep 07 03:55:43.134535 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:43.134456 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" event={"ID":"79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8","Type":"ContainerStarted","Data":"1a5b0be2e8c7036f3987fb4bfd6116fe95e20652f215406dfba4bb914962ec8d"} Sep 07 03:55:43.135003 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:43.134569 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" Sep 07 03:55:43.151390 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:43.151276 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" podStartSLOduration=1.882546149 podStartE2EDuration="3.151261615s" totalImagesPullingTime="1.268715466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:55:43.151164902 +0000 UTC m=+461.726031193" watchObservedRunningTime="2026-09-07 03:55:43.151261615 +0000 UTC m=+461.726127903" Sep 07 03:55:44.140583 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:44.140485 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8h7l" event={"ID":"c75f1de9-7bb2-4720-b121-bb7bb01998f0","Type":"ContainerStarted","Data":"b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954"} Sep 07 03:55:44.154510 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:44.154465 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-b8h7l" podStartSLOduration=2.029060925 podStartE2EDuration="4.154450565s" totalImagesPullingTime="2.12538964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:55:44.153941974 +0000 UTC m=+462.728808268" watchObservedRunningTime="2026-09-07 03:55:44.154450565 +0000 UTC m=+462.729316854" Sep 07 03:55:44.549661 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:44.549622 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8h7l"] Sep 07 03:55:46.151802 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:46.151764 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-b8h7l" podUID="c75f1de9-7bb2-4720-b121-bb7bb01998f0" containerName="authorino" containerID="cri-o://b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954" gracePeriod=30 Sep 07 03:55:46.384238 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:46.384215 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8h7l" Sep 07 03:55:46.405839 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:46.405788 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9r6c\" (UniqueName: \"kubernetes.io/projected/c75f1de9-7bb2-4720-b121-bb7bb01998f0-kube-api-access-h9r6c\") pod \"c75f1de9-7bb2-4720-b121-bb7bb01998f0\" (UID: \"c75f1de9-7bb2-4720-b121-bb7bb01998f0\") " Sep 07 03:55:46.407782 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:46.407759 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75f1de9-7bb2-4720-b121-bb7bb01998f0-kube-api-access-h9r6c" (OuterVolumeSpecName: "kube-api-access-h9r6c") pod "c75f1de9-7bb2-4720-b121-bb7bb01998f0" (UID: "c75f1de9-7bb2-4720-b121-bb7bb01998f0"). InnerVolumeSpecName "kube-api-access-h9r6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:55:46.506436 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:46.506410 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9r6c\" (UniqueName: \"kubernetes.io/projected/c75f1de9-7bb2-4720-b121-bb7bb01998f0-kube-api-access-h9r6c\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:55:47.156899 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:47.156862 2555 generic.go:356] "Generic (PLEG): container finished" podID="c75f1de9-7bb2-4720-b121-bb7bb01998f0" containerID="b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954" exitCode=0 Sep 07 03:55:47.156899 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:47.156901 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8h7l" Sep 07 03:55:47.157379 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:47.156943 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8h7l" event={"ID":"c75f1de9-7bb2-4720-b121-bb7bb01998f0","Type":"ContainerDied","Data":"b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954"} Sep 07 03:55:47.157379 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:47.156983 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8h7l" event={"ID":"c75f1de9-7bb2-4720-b121-bb7bb01998f0","Type":"ContainerDied","Data":"a4bdcd2b3bb00b50843dbf1ffd96e4c316e81c5d62e2cd2f2a099035ce7cd860"} Sep 07 03:55:47.157379 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:47.157001 2555 scope.go:117] "RemoveContainer" containerID="b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954" Sep 07 03:55:47.165303 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:47.165287 2555 scope.go:117] "RemoveContainer" containerID="b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954" Sep 07 03:55:47.165572 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:55:47.165555 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954\": container with ID starting with b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954 not found: ID does not exist" containerID="b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954" Sep 07 03:55:47.165613 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:47.165581 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954"} err="failed to get container status \"b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954\": rpc error: code = NotFound desc = could not find container \"b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954\": container with ID starting with b217e636871b76baa63f45c093a717e7337aae9e7957b4b2e18a8279a3885954 not found: ID does not exist" Sep 07 03:55:47.177159 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:47.177137 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8h7l"] Sep 07 03:55:47.180958 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:47.180940 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8h7l"] Sep 07 03:55:47.935829 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:47.935795 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c75f1de9-7bb2-4720-b121-bb7bb01998f0" path="/var/lib/kubelet/pods/c75f1de9-7bb2-4720-b121-bb7bb01998f0/volumes" Sep 07 03:55:54.141684 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:55:54.141643 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7tl54" Sep 07 03:56:09.206774 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.206745 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-d62qq"] Sep 07 03:56:09.207147 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.207133 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c75f1de9-7bb2-4720-b121-bb7bb01998f0" containerName="authorino" Sep 07 03:56:09.207196 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.207149 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c75f1de9-7bb2-4720-b121-bb7bb01998f0" containerName="authorino" Sep 07 03:56:09.207242 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.207230 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c75f1de9-7bb2-4720-b121-bb7bb01998f0" containerName="authorino" Sep 07 03:56:09.211704 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.211684 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" Sep 07 03:56:09.214926 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.214896 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mws5h\"" Sep 07 03:56:09.216682 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.216660 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-d62qq"] Sep 07 03:56:09.287503 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.287471 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4t4\" (UniqueName: \"kubernetes.io/projected/560bd0da-a96d-40d6-89a9-0a0853112398-kube-api-access-cw4t4\") pod \"authorino-5dc7ccb5f-d62qq\" (UID: \"560bd0da-a96d-40d6-89a9-0a0853112398\") " pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" Sep 07 03:56:09.388203 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.388174 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4t4\" (UniqueName: \"kubernetes.io/projected/560bd0da-a96d-40d6-89a9-0a0853112398-kube-api-access-cw4t4\") pod \"authorino-5dc7ccb5f-d62qq\" (UID: \"560bd0da-a96d-40d6-89a9-0a0853112398\") " pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" Sep 07 03:56:09.398451 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.398425 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4t4\" (UniqueName: \"kubernetes.io/projected/560bd0da-a96d-40d6-89a9-0a0853112398-kube-api-access-cw4t4\") pod \"authorino-5dc7ccb5f-d62qq\" (UID: \"560bd0da-a96d-40d6-89a9-0a0853112398\") " pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" Sep 07 03:56:09.409861 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.409838 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-d62qq"] Sep 07 03:56:09.410049 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.410038 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" Sep 07 03:56:09.436154 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.436130 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-sbskk"] Sep 07 03:56:09.440390 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.440371 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-sbskk" Sep 07 03:56:09.442891 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.442870 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 07 03:56:09.446305 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.446283 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-sbskk"] Sep 07 03:56:09.487275 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.487246 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-sbskk"] Sep 07 03:56:09.487531 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:56:09.487510 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-kxk25 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-kxk25 tls-cert]: context canceled" pod="kuadrant-system/authorino-687bd45585-sbskk" podUID="e653c0de-2b81-4c51-bb31-4daf6819e0a3" Sep 07 03:56:09.516413 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.516384 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65fff58977-lm2br"] Sep 07 03:56:09.520011 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.519995 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65fff58977-lm2br" Sep 07 03:56:09.527312 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.527290 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65fff58977-lm2br"] Sep 07 03:56:09.589351 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.589321 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dnbf\" (UniqueName: \"kubernetes.io/projected/d91fdc7f-825d-4c36-95d4-4025f625527f-kube-api-access-8dnbf\") pod \"authorino-65fff58977-lm2br\" (UID: \"d91fdc7f-825d-4c36-95d4-4025f625527f\") " pod="kuadrant-system/authorino-65fff58977-lm2br" Sep 07 03:56:09.594083 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.589835 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d91fdc7f-825d-4c36-95d4-4025f625527f-tls-cert\") pod \"authorino-65fff58977-lm2br\" (UID: \"d91fdc7f-825d-4c36-95d4-4025f625527f\") " pod="kuadrant-system/authorino-65fff58977-lm2br" Sep 07 03:56:09.594083 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.589941 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e653c0de-2b81-4c51-bb31-4daf6819e0a3-tls-cert\") pod \"authorino-687bd45585-sbskk\" (UID: \"e653c0de-2b81-4c51-bb31-4daf6819e0a3\") " pod="kuadrant-system/authorino-687bd45585-sbskk" Sep 07 03:56:09.594083 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.589984 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxk25\" (UniqueName: \"kubernetes.io/projected/e653c0de-2b81-4c51-bb31-4daf6819e0a3-kube-api-access-kxk25\") pod \"authorino-687bd45585-sbskk\" (UID: \"e653c0de-2b81-4c51-bb31-4daf6819e0a3\") " pod="kuadrant-system/authorino-687bd45585-sbskk" Sep 07 03:56:09.685796 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.685643 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-d62qq"] Sep 07 03:56:09.688706 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:56:09.688680 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod560bd0da_a96d_40d6_89a9_0a0853112398.slice/crio-594dd5550f5541aab1672f0fc9099bfb34b53325b870f108b3fb5a3334d262b2 WatchSource:0}: Error finding container 594dd5550f5541aab1672f0fc9099bfb34b53325b870f108b3fb5a3334d262b2: Status 404 returned error can't find the container with id 594dd5550f5541aab1672f0fc9099bfb34b53325b870f108b3fb5a3334d262b2 Sep 07 03:56:09.690495 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.690475 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dnbf\" (UniqueName: \"kubernetes.io/projected/d91fdc7f-825d-4c36-95d4-4025f625527f-kube-api-access-8dnbf\") pod \"authorino-65fff58977-lm2br\" (UID: \"d91fdc7f-825d-4c36-95d4-4025f625527f\") " pod="kuadrant-system/authorino-65fff58977-lm2br" Sep 07 03:56:09.690566 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.690508 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d91fdc7f-825d-4c36-95d4-4025f625527f-tls-cert\") pod \"authorino-65fff58977-lm2br\" (UID: \"d91fdc7f-825d-4c36-95d4-4025f625527f\") " pod="kuadrant-system/authorino-65fff58977-lm2br" Sep 07 03:56:09.690566 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.690542 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e653c0de-2b81-4c51-bb31-4daf6819e0a3-tls-cert\") pod \"authorino-687bd45585-sbskk\" (UID: \"e653c0de-2b81-4c51-bb31-4daf6819e0a3\") " pod="kuadrant-system/authorino-687bd45585-sbskk" Sep 07 03:56:09.690566 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.690565 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxk25\" (UniqueName: \"kubernetes.io/projected/e653c0de-2b81-4c51-bb31-4daf6819e0a3-kube-api-access-kxk25\") pod \"authorino-687bd45585-sbskk\" (UID: \"e653c0de-2b81-4c51-bb31-4daf6819e0a3\") " pod="kuadrant-system/authorino-687bd45585-sbskk" Sep 07 03:56:09.692927 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.692894 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e653c0de-2b81-4c51-bb31-4daf6819e0a3-tls-cert\") pod \"authorino-687bd45585-sbskk\" (UID: \"e653c0de-2b81-4c51-bb31-4daf6819e0a3\") " pod="kuadrant-system/authorino-687bd45585-sbskk" Sep 07 03:56:09.693243 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.693224 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d91fdc7f-825d-4c36-95d4-4025f625527f-tls-cert\") pod \"authorino-65fff58977-lm2br\" (UID: \"d91fdc7f-825d-4c36-95d4-4025f625527f\") " pod="kuadrant-system/authorino-65fff58977-lm2br" Sep 07 03:56:09.703798 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.703771 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxk25\" (UniqueName: \"kubernetes.io/projected/e653c0de-2b81-4c51-bb31-4daf6819e0a3-kube-api-access-kxk25\") pod \"authorino-687bd45585-sbskk\" (UID: \"e653c0de-2b81-4c51-bb31-4daf6819e0a3\") " pod="kuadrant-system/authorino-687bd45585-sbskk" Sep 07 03:56:09.704127 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.704112 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dnbf\" (UniqueName: \"kubernetes.io/projected/d91fdc7f-825d-4c36-95d4-4025f625527f-kube-api-access-8dnbf\") pod \"authorino-65fff58977-lm2br\" (UID: \"d91fdc7f-825d-4c36-95d4-4025f625527f\") " pod="kuadrant-system/authorino-65fff58977-lm2br" Sep 07 03:56:09.830204 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:09.830152 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65fff58977-lm2br" Sep 07 03:56:10.098306 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.098284 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65fff58977-lm2br"] Sep 07 03:56:10.100520 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:56:10.100490 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd91fdc7f_825d_4c36_95d4_4025f625527f.slice/crio-d58bfe3c3a75caa2961777b1e5a1fda162dab622e09f4c08ff6401cbda2db06b WatchSource:0}: Error finding container d58bfe3c3a75caa2961777b1e5a1fda162dab622e09f4c08ff6401cbda2db06b: Status 404 returned error can't find the container with id d58bfe3c3a75caa2961777b1e5a1fda162dab622e09f4c08ff6401cbda2db06b Sep 07 03:56:10.265552 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.265520 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65fff58977-lm2br" event={"ID":"d91fdc7f-825d-4c36-95d4-4025f625527f","Type":"ContainerStarted","Data":"d58bfe3c3a75caa2961777b1e5a1fda162dab622e09f4c08ff6401cbda2db06b"} Sep 07 03:56:10.267146 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.267120 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" event={"ID":"560bd0da-a96d-40d6-89a9-0a0853112398","Type":"ContainerStarted","Data":"99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2"} Sep 07 03:56:10.267279 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.267154 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" event={"ID":"560bd0da-a96d-40d6-89a9-0a0853112398","Type":"ContainerStarted","Data":"594dd5550f5541aab1672f0fc9099bfb34b53325b870f108b3fb5a3334d262b2"} Sep 07 03:56:10.267279 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.267174 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-sbskk" Sep 07 03:56:10.267279 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.267167 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" podUID="560bd0da-a96d-40d6-89a9-0a0853112398" containerName="authorino" containerID="cri-o://99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2" gracePeriod=30 Sep 07 03:56:10.274560 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.274541 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-sbskk" Sep 07 03:56:10.282442 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.282402 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" podStartSLOduration=0.909184322 podStartE2EDuration="1.282388376s" totalImagesPullingTime="373.204054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:56:10.280884001 +0000 UTC m=+488.855750293" watchObservedRunningTime="2026-09-07 03:56:10.282388376 +0000 UTC m=+488.857254665" Sep 07 03:56:10.397778 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.397753 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e653c0de-2b81-4c51-bb31-4daf6819e0a3-tls-cert\") pod \"e653c0de-2b81-4c51-bb31-4daf6819e0a3\" (UID: \"e653c0de-2b81-4c51-bb31-4daf6819e0a3\") " Sep 07 03:56:10.397884 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.397812 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxk25\" (UniqueName: \"kubernetes.io/projected/e653c0de-2b81-4c51-bb31-4daf6819e0a3-kube-api-access-kxk25\") pod \"e653c0de-2b81-4c51-bb31-4daf6819e0a3\" (UID: \"e653c0de-2b81-4c51-bb31-4daf6819e0a3\") " Sep 07 03:56:10.399760 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.399727 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e653c0de-2b81-4c51-bb31-4daf6819e0a3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e653c0de-2b81-4c51-bb31-4daf6819e0a3" (UID: "e653c0de-2b81-4c51-bb31-4daf6819e0a3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 03:56:10.399884 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.399862 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e653c0de-2b81-4c51-bb31-4daf6819e0a3-kube-api-access-kxk25" (OuterVolumeSpecName: "kube-api-access-kxk25") pod "e653c0de-2b81-4c51-bb31-4daf6819e0a3" (UID: "e653c0de-2b81-4c51-bb31-4daf6819e0a3"). InnerVolumeSpecName "kube-api-access-kxk25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:56:10.498625 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.498604 2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e653c0de-2b81-4c51-bb31-4daf6819e0a3-tls-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:56:10.498714 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.498629 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxk25\" (UniqueName: \"kubernetes.io/projected/e653c0de-2b81-4c51-bb31-4daf6819e0a3-kube-api-access-kxk25\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:56:10.535801 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.535778 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" Sep 07 03:56:10.599942 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.599908 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw4t4\" (UniqueName: \"kubernetes.io/projected/560bd0da-a96d-40d6-89a9-0a0853112398-kube-api-access-cw4t4\") pod \"560bd0da-a96d-40d6-89a9-0a0853112398\" (UID: \"560bd0da-a96d-40d6-89a9-0a0853112398\") " Sep 07 03:56:10.601965 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.601940 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560bd0da-a96d-40d6-89a9-0a0853112398-kube-api-access-cw4t4" (OuterVolumeSpecName: "kube-api-access-cw4t4") pod "560bd0da-a96d-40d6-89a9-0a0853112398" (UID: "560bd0da-a96d-40d6-89a9-0a0853112398"). InnerVolumeSpecName "kube-api-access-cw4t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:56:10.701220 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:10.701198 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw4t4\" (UniqueName: \"kubernetes.io/projected/560bd0da-a96d-40d6-89a9-0a0853112398-kube-api-access-cw4t4\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:56:11.273560 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.273524 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65fff58977-lm2br" event={"ID":"d91fdc7f-825d-4c36-95d4-4025f625527f","Type":"ContainerStarted","Data":"0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10"} Sep 07 03:56:11.274751 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.274719 2555 generic.go:356] "Generic (PLEG): container finished" podID="560bd0da-a96d-40d6-89a9-0a0853112398" containerID="99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2" exitCode=0 Sep 07 03:56:11.274751 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.274748 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" Sep 07 03:56:11.275066 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.274793 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" event={"ID":"560bd0da-a96d-40d6-89a9-0a0853112398","Type":"ContainerDied","Data":"99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2"} Sep 07 03:56:11.275066 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.274825 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-d62qq" event={"ID":"560bd0da-a96d-40d6-89a9-0a0853112398","Type":"ContainerDied","Data":"594dd5550f5541aab1672f0fc9099bfb34b53325b870f108b3fb5a3334d262b2"} Sep 07 03:56:11.275066 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.274843 2555 scope.go:117] "RemoveContainer" containerID="99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2" Sep 07 03:56:11.275066 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.274966 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-sbskk" Sep 07 03:56:11.283091 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.283074 2555 scope.go:117] "RemoveContainer" containerID="99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2" Sep 07 03:56:11.283345 ip-10-0-131-215 kubenswrapper[2555]: E0907 03:56:11.283328 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2\": container with ID starting with 99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2 not found: ID does not exist" containerID="99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2" Sep 07 03:56:11.283416 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.283352 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2"} err="failed to get container status \"99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2\": rpc error: code = NotFound desc = could not find container \"99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2\": container with ID starting with 99b0a5ae58d72b3d1ebc4fb6dfcedc169ed206c82fcc9ba30e921d25b24d0df2 not found: ID does not exist" Sep 07 03:56:11.289300 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.289263 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65fff58977-lm2br" podStartSLOduration=1.901499302 podStartE2EDuration="2.289250361s" totalImagesPullingTime="387.751059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:56:11.288551844 +0000 UTC m=+489.863418134" watchObservedRunningTime="2026-09-07 03:56:11.289250361 +0000 UTC m=+489.864116671" Sep 07 03:56:11.314082 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.314045 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-sbskk"] Sep 07 03:56:11.320357 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.320332 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-sbskk"] Sep 07 03:56:11.333016 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.332990 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-d62qq"] Sep 07 03:56:11.342278 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.342253 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-d62qq"] Sep 07 03:56:11.936156 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.936118 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560bd0da-a96d-40d6-89a9-0a0853112398" path="/var/lib/kubelet/pods/560bd0da-a96d-40d6-89a9-0a0853112398/volumes" Sep 07 03:56:11.936557 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:11.936535 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e653c0de-2b81-4c51-bb31-4daf6819e0a3" path="/var/lib/kubelet/pods/e653c0de-2b81-4c51-bb31-4daf6819e0a3/volumes" Sep 07 03:56:54.191367 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:56:54.191285 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:57:03.525235 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:03.525201 2555 generic.go:356] "Generic (PLEG): container finished" podID="ab9bed40-0504-4031-b120-d3d199e315a6" containerID="77881066059cf2313bb0980e8a3b3524bea6a4693c3468d356202d68c451ccdc" exitCode=0 Sep 07 03:57:03.525726 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:03.525265 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" event={"ID":"ab9bed40-0504-4031-b120-d3d199e315a6","Type":"ContainerDied","Data":"77881066059cf2313bb0980e8a3b3524bea6a4693c3468d356202d68c451ccdc"} Sep 07 03:57:03.525726 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:03.525707 2555 scope.go:117] "RemoveContainer" containerID="77881066059cf2313bb0980e8a3b3524bea6a4693c3468d356202d68c451ccdc" Sep 07 03:57:03.526698 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:03.526669 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 03:57:03.526796 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:03.526749 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 03:57:03.526855 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:03.526794 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 03:57:04.531514 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:04.531477 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" event={"ID":"ab9bed40-0504-4031-b120-d3d199e315a6","Type":"ContainerStarted","Data":"8e5872ac23983608e05258ea3bb1d33eb16ba46ddd870ea825379adf4e95c563"} Sep 07 03:57:04.568520 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:04.568483 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:57:04.568728 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:04.568613 2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-d7pgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 03:57:04.568728 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:04.568671 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" podUID="ab9bed40-0504-4031-b120-d3d199e315a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 03:57:05.568499 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:05.568461 2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-d7pgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 03:57:05.568859 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:05.568530 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" podUID="ab9bed40-0504-4031-b120-d3d199e315a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 03:57:06.567877 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:06.567839 2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-d7pgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 03:57:06.568052 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:06.567908 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" podUID="ab9bed40-0504-4031-b120-d3d199e315a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 03:57:07.568553 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:07.568522 2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-d7pgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 03:57:07.569002 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:07.568594 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" podUID="ab9bed40-0504-4031-b120-d3d199e315a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 03:57:08.568063 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:08.568032 2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-d7pgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 03:57:08.568253 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:08.568119 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" podUID="ab9bed40-0504-4031-b120-d3d199e315a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 03:57:09.568346 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:09.568311 2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-d7pgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 03:57:09.568720 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:09.568360 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" podUID="ab9bed40-0504-4031-b120-d3d199e315a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 03:57:10.113364 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:10.113327 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:57:10.571359 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:10.571330 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:57:10.572260 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:10.572241 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-d7pgb" Sep 07 03:57:39.890797 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:39.890753 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:57:54.328646 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:57:54.328606 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:58:00.073018 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.072977 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n"] Sep 07 03:58:00.073549 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.073533 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560bd0da-a96d-40d6-89a9-0a0853112398" containerName="authorino" Sep 07 03:58:00.073592 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.073554 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="560bd0da-a96d-40d6-89a9-0a0853112398" containerName="authorino" Sep 07 03:58:00.073658 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.073648 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="560bd0da-a96d-40d6-89a9-0a0853112398" containerName="authorino" Sep 07 03:58:00.077220 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.077198 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.080774 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.080740 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v92z7\"" Sep 07 03:58:00.080774 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.080747 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 07 03:58:00.080958 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.080747 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 07 03:58:00.080958 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.080820 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 07 03:58:00.084787 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.084641 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n"] Sep 07 03:58:00.194585 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.194542 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8c9j\" (UniqueName: \"kubernetes.io/projected/8215cbec-b876-475d-b35f-afa868f2d6ef-kube-api-access-m8c9j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.194585 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.194590 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.194796 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.194621 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.194796 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.194641 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.194796 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.194679 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.194796 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.194720 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8215cbec-b876-475d-b35f-afa868f2d6ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.296141 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.296104 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8c9j\" (UniqueName: \"kubernetes.io/projected/8215cbec-b876-475d-b35f-afa868f2d6ef-kube-api-access-m8c9j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.296141 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.296145 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.296379 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.296166 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.296379 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.296183 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.296379 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.296233 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.296379 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.296268 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8215cbec-b876-475d-b35f-afa868f2d6ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.296630 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.296584 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.296688 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.296629 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.296746 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.296700 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.298454 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.298428 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8215cbec-b876-475d-b35f-afa868f2d6ef-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.298773 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.298756 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8215cbec-b876-475d-b35f-afa868f2d6ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.304436 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.304412 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8c9j\" (UniqueName: \"kubernetes.io/projected/8215cbec-b876-475d-b35f-afa868f2d6ef-kube-api-access-m8c9j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n\" (UID: \"8215cbec-b876-475d-b35f-afa868f2d6ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.388307 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.388217 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:00.663363 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.663287 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n"] Sep 07 03:58:00.667393 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:58:00.667364 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8215cbec_b876_475d_b35f_afa868f2d6ef.slice/crio-da41ee518354edb9b7e9191e43ddefd213fda7ef386293aa3a55803007673458 WatchSource:0}: Error finding container da41ee518354edb9b7e9191e43ddefd213fda7ef386293aa3a55803007673458: Status 404 returned error can't find the container with id da41ee518354edb9b7e9191e43ddefd213fda7ef386293aa3a55803007673458 Sep 07 03:58:00.803639 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:00.803600 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" event={"ID":"8215cbec-b876-475d-b35f-afa868f2d6ef","Type":"ContainerStarted","Data":"da41ee518354edb9b7e9191e43ddefd213fda7ef386293aa3a55803007673458"} Sep 07 03:58:01.293861 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:01.293825 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:58:01.811626 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:01.811587 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" event={"ID":"8215cbec-b876-475d-b35f-afa868f2d6ef","Type":"ContainerStarted","Data":"b0e9006cd821c553241d9c6ba31e3de5b6dc43072b931267d83e2820778ea5d4"} Sep 07 03:58:01.833118 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:01.833036 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" podStartSLOduration=0.857276734 podStartE2EDuration="1.833018035s" totalImagesPullingTime="975.741301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:58:01.830885359 +0000 UTC m=+600.405751650" watchObservedRunningTime="2026-09-07 03:58:01.833018035 +0000 UTC m=+600.407884325" Sep 07 03:58:01.894602 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:01.894511 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 03:58:01.895285 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:01.895257 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 03:58:07.568591 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.568556 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv"] Sep 07 03:58:07.572613 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.572595 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.575068 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.575034 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 07 03:58:07.582766 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.582739 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv"] Sep 07 03:58:07.663019 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.662985 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff86904a-3699-4173-bbf2-7786d666ff9f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.663019 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.663020 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.663019 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.663039 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.663290 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.663205 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.663290 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.663236 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.663290 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.663256 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq57l\" (UniqueName: \"kubernetes.io/projected/ff86904a-3699-4173-bbf2-7786d666ff9f-kube-api-access-qq57l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.764727 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.764690 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.764727 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.764728 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq57l\" (UniqueName: \"kubernetes.io/projected/ff86904a-3699-4173-bbf2-7786d666ff9f-kube-api-access-qq57l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.764961 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.764797 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff86904a-3699-4173-bbf2-7786d666ff9f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.764961 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.764823 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.764961 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.764933 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.765149 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.765021 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.765206 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.765154 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.765359 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.765334 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.765436 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.765380 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.767149 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.767124 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff86904a-3699-4173-bbf2-7786d666ff9f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.767331 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.767316 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff86904a-3699-4173-bbf2-7786d666ff9f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.772976 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.772955 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq57l\" (UniqueName: \"kubernetes.io/projected/ff86904a-3699-4173-bbf2-7786d666ff9f-kube-api-access-qq57l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ddwhv\" (UID: \"ff86904a-3699-4173-bbf2-7786d666ff9f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:07.885709 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:07.885624 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:08.156437 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:08.156413 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv"] Sep 07 03:58:08.158209 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:58:08.158181 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff86904a_3699_4173_bbf2_7786d666ff9f.slice/crio-a18c148eeea5ba01d5d9da8e961158e29b7cc2c765e5e6875886319bace88615 WatchSource:0}: Error finding container a18c148eeea5ba01d5d9da8e961158e29b7cc2c765e5e6875886319bace88615: Status 404 returned error can't find the container with id a18c148eeea5ba01d5d9da8e961158e29b7cc2c765e5e6875886319bace88615 Sep 07 03:58:08.792391 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:08.792358 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:58:08.847929 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:08.847837 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" event={"ID":"ff86904a-3699-4173-bbf2-7786d666ff9f","Type":"ContainerStarted","Data":"360992902ee227995bb7aae609bb480cc0b8053215bf6089be67ca587f54a276"} Sep 07 03:58:08.848196 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:08.847959 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" event={"ID":"ff86904a-3699-4173-bbf2-7786d666ff9f","Type":"ContainerStarted","Data":"a18c148eeea5ba01d5d9da8e961158e29b7cc2c765e5e6875886319bace88615"} Sep 07 03:58:08.864881 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:08.864820 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" podStartSLOduration=1.6693491809999998 podStartE2EDuration="1.864799275s" totalImagesPullingTime="195.450094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:58:08.864732281 +0000 UTC m=+607.439598572" watchObservedRunningTime="2026-09-07 03:58:08.864799275 +0000 UTC m=+607.439665566" Sep 07 03:58:10.388618 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:10.388580 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:10.389126 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:10.388650 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:10.401074 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:10.401033 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:10.870573 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:10.870543 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n" Sep 07 03:58:14.379299 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.379265 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d"] Sep 07 03:58:14.383304 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.383285 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.385652 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.385624 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 07 03:58:14.392181 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.392156 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d"] Sep 07 03:58:14.525039 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.525006 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.525219 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.525042 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8cwb\" (UniqueName: \"kubernetes.io/projected/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-kube-api-access-v8cwb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.525219 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.525149 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.525219 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.525187 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.525383 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.525274 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.525383 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.525321 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.626404 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.626357 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.626594 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.626419 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.626594 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.626488 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.626594 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.626513 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8cwb\" (UniqueName: \"kubernetes.io/projected/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-kube-api-access-v8cwb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.626594 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.626543 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.626594 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.626574 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.626892 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.626862 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.627012 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.626858 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.627012 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.626956 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.628640 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.628624 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.628913 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.628893 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.634456 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.634402 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8cwb\" (UniqueName: \"kubernetes.io/projected/97920ef3-0c7c-438b-9508-3d3b0ddd0c92-kube-api-access-v8cwb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d\" (UID: \"97920ef3-0c7c-438b-9508-3d3b0ddd0c92\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.695408 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.695381 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:14.965673 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:14.965647 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d"] Sep 07 03:58:14.968097 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:58:14.968068 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97920ef3_0c7c_438b_9508_3d3b0ddd0c92.slice/crio-25617dd003e088fd2f94d2f1780aba90701ac9278b7d75b36bbbada598aaa9af WatchSource:0}: Error finding container 25617dd003e088fd2f94d2f1780aba90701ac9278b7d75b36bbbada598aaa9af: Status 404 returned error can't find the container with id 25617dd003e088fd2f94d2f1780aba90701ac9278b7d75b36bbbada598aaa9af Sep 07 03:58:15.394445 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:15.394366 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:58:15.885647 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:15.885607 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" event={"ID":"97920ef3-0c7c-438b-9508-3d3b0ddd0c92","Type":"ContainerStarted","Data":"d32ebc2dbe179a2086a040b7f09b4786e3210a9bf9cfed6a651014730f4aab80"} Sep 07 03:58:15.885844 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:15.885675 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" event={"ID":"97920ef3-0c7c-438b-9508-3d3b0ddd0c92","Type":"ContainerStarted","Data":"25617dd003e088fd2f94d2f1780aba90701ac9278b7d75b36bbbada598aaa9af"} Sep 07 03:58:15.905594 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:15.905542 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" podStartSLOduration=1.727268036 podStartE2EDuration="1.905527285s" totalImagesPullingTime="178.259249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:58:15.903865263 +0000 UTC m=+614.478731557" watchObservedRunningTime="2026-09-07 03:58:15.905527285 +0000 UTC m=+614.480393588" Sep 07 03:58:17.886557 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:17.886515 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:17.886930 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:17.886580 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:17.899365 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:17.899336 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:17.910602 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:17.910580 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ddwhv" Sep 07 03:58:24.696039 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:24.695954 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:24.696039 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:24.696032 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:24.708244 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:24.708221 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:24.941851 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:24.941815 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d" Sep 07 03:58:25.004704 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:25.004666 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:58:38.896301 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:58:38.896268 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 03:59:57.874368 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:57.874278 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bd9996b56-wm7mx"] Sep 07 03:59:57.877837 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:57.877817 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd9996b56-wm7mx" Sep 07 03:59:57.885169 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:57.885148 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bd9996b56-wm7mx"] Sep 07 03:59:58.003323 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:58.003281 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bk2\" (UniqueName: \"kubernetes.io/projected/53e18245-6542-4222-8d9a-ced383830854-kube-api-access-k4bk2\") pod \"authorino-7bd9996b56-wm7mx\" (UID: \"53e18245-6542-4222-8d9a-ced383830854\") " pod="kuadrant-system/authorino-7bd9996b56-wm7mx" Sep 07 03:59:58.003323 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:58.003328 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/53e18245-6542-4222-8d9a-ced383830854-tls-cert\") pod \"authorino-7bd9996b56-wm7mx\" (UID: \"53e18245-6542-4222-8d9a-ced383830854\") " pod="kuadrant-system/authorino-7bd9996b56-wm7mx" Sep 07 03:59:58.104170 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:58.104128 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bk2\" (UniqueName: \"kubernetes.io/projected/53e18245-6542-4222-8d9a-ced383830854-kube-api-access-k4bk2\") pod \"authorino-7bd9996b56-wm7mx\" (UID: \"53e18245-6542-4222-8d9a-ced383830854\") " pod="kuadrant-system/authorino-7bd9996b56-wm7mx" Sep 07 03:59:58.104349 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:58.104195 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/53e18245-6542-4222-8d9a-ced383830854-tls-cert\") pod \"authorino-7bd9996b56-wm7mx\" (UID: \"53e18245-6542-4222-8d9a-ced383830854\") " pod="kuadrant-system/authorino-7bd9996b56-wm7mx" Sep 07 03:59:58.106647 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:58.106625 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/53e18245-6542-4222-8d9a-ced383830854-tls-cert\") pod \"authorino-7bd9996b56-wm7mx\" (UID: \"53e18245-6542-4222-8d9a-ced383830854\") " pod="kuadrant-system/authorino-7bd9996b56-wm7mx" Sep 07 03:59:58.111582 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:58.111558 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bk2\" (UniqueName: \"kubernetes.io/projected/53e18245-6542-4222-8d9a-ced383830854-kube-api-access-k4bk2\") pod \"authorino-7bd9996b56-wm7mx\" (UID: \"53e18245-6542-4222-8d9a-ced383830854\") " pod="kuadrant-system/authorino-7bd9996b56-wm7mx" Sep 07 03:59:58.188066 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:58.187974 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd9996b56-wm7mx" Sep 07 03:59:58.457118 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:58.457091 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bd9996b56-wm7mx"] Sep 07 03:59:58.459093 ip-10-0-131-215 kubenswrapper[2555]: W0907 03:59:58.459035 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53e18245_6542_4222_8d9a_ced383830854.slice/crio-1b730842da88547f3cd3ac8e55324d70c9940ca949a194eb710c75b2fae88a58 WatchSource:0}: Error finding container 1b730842da88547f3cd3ac8e55324d70c9940ca949a194eb710c75b2fae88a58: Status 404 returned error can't find the container with id 1b730842da88547f3cd3ac8e55324d70c9940ca949a194eb710c75b2fae88a58 Sep 07 03:59:58.460414 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:58.460390 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 03:59:59.387667 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.387634 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bd9996b56-wm7mx" event={"ID":"53e18245-6542-4222-8d9a-ced383830854","Type":"ContainerStarted","Data":"8d5141fa8c0f775d081aabeac7c565783e6704700c0fb062efe8b0c50712d865"} Sep 07 03:59:59.388014 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.387677 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bd9996b56-wm7mx" event={"ID":"53e18245-6542-4222-8d9a-ced383830854","Type":"ContainerStarted","Data":"1b730842da88547f3cd3ac8e55324d70c9940ca949a194eb710c75b2fae88a58"} Sep 07 03:59:59.405065 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.405004 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bd9996b56-wm7mx" podStartSLOduration=1.659364723 podStartE2EDuration="2.404979055s" totalImagesPullingTime="745.614332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 03:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 03:59:59.402900295 +0000 UTC m=+717.977766585" watchObservedRunningTime="2026-09-07 03:59:59.404979055 +0000 UTC m=+717.979845346" Sep 07 03:59:59.429543 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.429483 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65fff58977-lm2br"] Sep 07 03:59:59.429751 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.429718 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-65fff58977-lm2br" podUID="d91fdc7f-825d-4c36-95d4-4025f625527f" containerName="authorino" containerID="cri-o://0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10" gracePeriod=30 Sep 07 03:59:59.693433 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.693405 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65fff58977-lm2br" Sep 07 03:59:59.818567 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.818491 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dnbf\" (UniqueName: \"kubernetes.io/projected/d91fdc7f-825d-4c36-95d4-4025f625527f-kube-api-access-8dnbf\") pod \"d91fdc7f-825d-4c36-95d4-4025f625527f\" (UID: \"d91fdc7f-825d-4c36-95d4-4025f625527f\") " Sep 07 03:59:59.818721 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.818598 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d91fdc7f-825d-4c36-95d4-4025f625527f-tls-cert\") pod \"d91fdc7f-825d-4c36-95d4-4025f625527f\" (UID: \"d91fdc7f-825d-4c36-95d4-4025f625527f\") " Sep 07 03:59:59.820711 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.820668 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91fdc7f-825d-4c36-95d4-4025f625527f-kube-api-access-8dnbf" (OuterVolumeSpecName: "kube-api-access-8dnbf") pod "d91fdc7f-825d-4c36-95d4-4025f625527f" (UID: "d91fdc7f-825d-4c36-95d4-4025f625527f"). InnerVolumeSpecName "kube-api-access-8dnbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 03:59:59.829881 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.829851 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91fdc7f-825d-4c36-95d4-4025f625527f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d91fdc7f-825d-4c36-95d4-4025f625527f" (UID: "d91fdc7f-825d-4c36-95d4-4025f625527f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 03:59:59.919292 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.919258 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dnbf\" (UniqueName: \"kubernetes.io/projected/d91fdc7f-825d-4c36-95d4-4025f625527f-kube-api-access-8dnbf\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 03:59:59.919292 ip-10-0-131-215 kubenswrapper[2555]: I0907 03:59:59.919286 2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d91fdc7f-825d-4c36-95d4-4025f625527f-tls-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:00:00.394772 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:00.394726 2555 generic.go:356] "Generic (PLEG): container finished" podID="d91fdc7f-825d-4c36-95d4-4025f625527f" containerID="0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10" exitCode=0 Sep 07 04:00:00.394772 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:00.394762 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65fff58977-lm2br" Sep 07 04:00:00.395298 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:00.394804 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65fff58977-lm2br" event={"ID":"d91fdc7f-825d-4c36-95d4-4025f625527f","Type":"ContainerDied","Data":"0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10"} Sep 07 04:00:00.395298 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:00.394849 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65fff58977-lm2br" event={"ID":"d91fdc7f-825d-4c36-95d4-4025f625527f","Type":"ContainerDied","Data":"d58bfe3c3a75caa2961777b1e5a1fda162dab622e09f4c08ff6401cbda2db06b"} Sep 07 04:00:00.395298 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:00.394872 2555 scope.go:117] "RemoveContainer" containerID="0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10" Sep 07 04:00:00.404117 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:00.404098 2555 scope.go:117] "RemoveContainer" containerID="0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10" Sep 07 04:00:00.404406 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:00:00.404387 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10\": container with ID starting with 0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10 not found: ID does not exist" containerID="0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10" Sep 07 04:00:00.404464 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:00.404413 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10"} err="failed to get container status \"0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10\": rpc error: code = NotFound desc = could not find container \"0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10\": container with ID starting with 0022f016ce7bb641d9c373f28943696c21a68ca343eed4fb52b99e402c831f10 not found: ID does not exist" Sep 07 04:00:00.412925 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:00.412898 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65fff58977-lm2br"] Sep 07 04:00:00.421757 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:00.421730 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65fff58977-lm2br"] Sep 07 04:00:01.938539 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:01.938506 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d91fdc7f-825d-4c36-95d4-4025f625527f" path="/var/lib/kubelet/pods/d91fdc7f-825d-4c36-95d4-4025f625527f/volumes" Sep 07 04:00:19.988553 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:19.988516 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:00:22.185908 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.185864 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7365-139773466596400"] Sep 07 04:00:22.186408 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.186273 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91fdc7f-825d-4c36-95d4-4025f625527f" containerName="authorino" Sep 07 04:00:22.186408 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.186287 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91fdc7f-825d-4c36-95d4-4025f625527f" containerName="authorino" Sep 07 04:00:22.186408 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.186354 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d91fdc7f-825d-4c36-95d4-4025f625527f" containerName="authorino" Sep 07 04:00:22.190613 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.190591 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7365-139773466596400" Sep 07 04:00:22.193290 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.193271 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhk6n\"" Sep 07 04:00:22.196919 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.196901 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7365-139773466596400"] Sep 07 04:00:22.316266 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.316225 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fj2g\" (UniqueName: \"kubernetes.io/projected/b70f97e5-3554-4933-b3d8-6b6b97cb53d5-kube-api-access-8fj2g\") pod \"test-curl-7365-139773466596400\" (UID: \"b70f97e5-3554-4933-b3d8-6b6b97cb53d5\") " pod="opendatahub/test-curl-7365-139773466596400" Sep 07 04:00:22.417428 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.417394 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fj2g\" (UniqueName: \"kubernetes.io/projected/b70f97e5-3554-4933-b3d8-6b6b97cb53d5-kube-api-access-8fj2g\") pod \"test-curl-7365-139773466596400\" (UID: \"b70f97e5-3554-4933-b3d8-6b6b97cb53d5\") " pod="opendatahub/test-curl-7365-139773466596400" Sep 07 04:00:22.425215 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.425189 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fj2g\" (UniqueName: \"kubernetes.io/projected/b70f97e5-3554-4933-b3d8-6b6b97cb53d5-kube-api-access-8fj2g\") pod \"test-curl-7365-139773466596400\" (UID: \"b70f97e5-3554-4933-b3d8-6b6b97cb53d5\") " pod="opendatahub/test-curl-7365-139773466596400" Sep 07 04:00:22.501397 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.501372 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7365-139773466596400" Sep 07 04:00:22.773020 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:22.772997 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7365-139773466596400"] Sep 07 04:00:22.775640 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:00:22.775608 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb70f97e5_3554_4933_b3d8_6b6b97cb53d5.slice/crio-eda5fa74e8a3375f298f12d783774afe490b200e2e6fb90bfae0a8caeb577d52 WatchSource:0}: Error finding container eda5fa74e8a3375f298f12d783774afe490b200e2e6fb90bfae0a8caeb577d52: Status 404 returned error can't find the container with id eda5fa74e8a3375f298f12d783774afe490b200e2e6fb90bfae0a8caeb577d52 Sep 07 04:00:23.511370 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:23.511330 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7365-139773466596400" event={"ID":"b70f97e5-3554-4933-b3d8-6b6b97cb53d5","Type":"ContainerStarted","Data":"eda5fa74e8a3375f298f12d783774afe490b200e2e6fb90bfae0a8caeb577d52"} Sep 07 04:00:25.524953 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:25.524920 2555 generic.go:356] "Generic (PLEG): container finished" podID="b70f97e5-3554-4933-b3d8-6b6b97cb53d5" containerID="9d3350f404faaffc74b01b64c642bcb65ef604cb3f29d5db19dd2f237fc2163a" exitCode=0 Sep 07 04:00:25.525358 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:25.524966 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7365-139773466596400" event={"ID":"b70f97e5-3554-4933-b3d8-6b6b97cb53d5","Type":"ContainerDied","Data":"9d3350f404faaffc74b01b64c642bcb65ef604cb3f29d5db19dd2f237fc2163a"} Sep 07 04:00:26.679230 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:26.679202 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7365-139773466596400" Sep 07 04:00:26.713931 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:26.713907 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7365-139773466596400_b70f97e5-3554-4933-b3d8-6b6b97cb53d5/test-curl-7365-139773466596400/0.log" Sep 07 04:00:26.732662 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:26.732638 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7365-139773466596400"] Sep 07 04:00:26.735270 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:26.735243 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7365-139773466596400"] Sep 07 04:00:26.859168 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:26.859043 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fj2g\" (UniqueName: \"kubernetes.io/projected/b70f97e5-3554-4933-b3d8-6b6b97cb53d5-kube-api-access-8fj2g\") pod \"b70f97e5-3554-4933-b3d8-6b6b97cb53d5\" (UID: \"b70f97e5-3554-4933-b3d8-6b6b97cb53d5\") " Sep 07 04:00:26.861510 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:26.861479 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70f97e5-3554-4933-b3d8-6b6b97cb53d5-kube-api-access-8fj2g" (OuterVolumeSpecName: "kube-api-access-8fj2g") pod "b70f97e5-3554-4933-b3d8-6b6b97cb53d5" (UID: "b70f97e5-3554-4933-b3d8-6b6b97cb53d5"). InnerVolumeSpecName "kube-api-access-8fj2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:00:26.960138 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:26.960105 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fj2g\" (UniqueName: \"kubernetes.io/projected/b70f97e5-3554-4933-b3d8-6b6b97cb53d5-kube-api-access-8fj2g\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:00:27.554643 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.554604 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eda5fa74e8a3375f298f12d783774afe490b200e2e6fb90bfae0a8caeb577d52" Sep 07 04:00:27.554852 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.554694 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7365-139773466596400" Sep 07 04:00:27.555046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.555021 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x"] Sep 07 04:00:27.555731 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.555708 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70f97e5-3554-4933-b3d8-6b6b97cb53d5" containerName="test-curl-7365-139773466596400" Sep 07 04:00:27.555731 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.555731 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70f97e5-3554-4933-b3d8-6b6b97cb53d5" containerName="test-curl-7365-139773466596400" Sep 07 04:00:27.555912 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.555845 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b70f97e5-3554-4933-b3d8-6b6b97cb53d5" containerName="test-curl-7365-139773466596400" Sep 07 04:00:27.559176 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.559155 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.563523 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.563501 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-908959-openshift-default-dockercfg-tvw5s\"" Sep 07 04:00:27.568076 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.568037 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x"] Sep 07 04:00:27.582177 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:00:27.582135 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7365-139773466596400\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="b70f97e5-3554-4933-b3d8-6b6b97cb53d5" pod="opendatahub/test-curl-7365-139773466596400" Sep 07 04:00:27.666594 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.666556 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istiod-ca-cert\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.666781 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.666619 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-data\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.666781 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.666648 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-envoy\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.666781 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.666737 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-certs\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.666948 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.666810 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-credential-socket\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.666948 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.666844 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-podinfo\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.666948 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.666868 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzg8\" (UniqueName: \"kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-kube-api-access-5wzg8\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.666948 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.666910 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-token\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.667104 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.666968 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-socket\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.767602 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.767560 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-socket\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.767715 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istiod-ca-cert\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.767768 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-data\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.767816 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-envoy\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.767865 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-certs\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.767915 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-credential-socket\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.767945 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-podinfo\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.767971 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzg8\" (UniqueName: \"kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-kube-api-access-5wzg8\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.767999 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-token\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768427 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.768040 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-socket\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768427 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.768163 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-data\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768427 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.768348 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-credential-socket\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768427 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.768386 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istiod-ca-cert\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.768620 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.768591 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-certs\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.770489 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.770462 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-envoy\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.770751 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.770732 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-podinfo\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.776227 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.776198 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-token\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.776357 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.776329 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzg8\" (UniqueName: \"kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-kube-api-access-5wzg8\") pod \"e2e-autores-908959-openshift-default-f468d74f8-hl98x\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.877752 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.877662 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:27.946788 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:27.945753 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b70f97e5-3554-4933-b3d8-6b6b97cb53d5" path="/var/lib/kubelet/pods/b70f97e5-3554-4933-b3d8-6b6b97cb53d5/volumes" Sep 07 04:00:28.167550 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:28.167521 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x"] Sep 07 04:00:28.170011 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:00:28.169979 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97b3dc0_690b_4661_86b7_638f1e41cfb2.slice/crio-fdbdf265df846f2a23df660b804721c115d23cad707d6f9be9de4ca72e606dc7 WatchSource:0}: Error finding container fdbdf265df846f2a23df660b804721c115d23cad707d6f9be9de4ca72e606dc7: Status 404 returned error can't find the container with id fdbdf265df846f2a23df660b804721c115d23cad707d6f9be9de4ca72e606dc7 Sep 07 04:00:28.172863 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:28.172806 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:00:28.172954 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:28.172906 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:00:28.173014 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:28.172953 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:00:28.562129 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:28.562090 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" event={"ID":"b97b3dc0-690b-4661-86b7-638f1e41cfb2","Type":"ContainerStarted","Data":"ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e"} Sep 07 04:00:28.562304 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:28.562152 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" event={"ID":"b97b3dc0-690b-4661-86b7-638f1e41cfb2","Type":"ContainerStarted","Data":"fdbdf265df846f2a23df660b804721c115d23cad707d6f9be9de4ca72e606dc7"} Sep 07 04:00:28.582323 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:28.582277 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" podStartSLOduration=1.582261041 podStartE2EDuration="1.582261041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:00:28.580780136 +0000 UTC m=+747.155646427" watchObservedRunningTime="2026-09-07 04:00:28.582261041 +0000 UTC m=+747.157127330" Sep 07 04:00:28.878416 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:28.878332 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:29.884146 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:29.884114 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:30.572594 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:30.572564 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:30.573627 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:30.573608 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:32.994283 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:32.994248 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:00:36.864570 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:36.864534 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg"] Sep 07 04:00:36.869297 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:36.869273 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:36.871957 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:36.871933 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-9065bcd9-kserve-self-signed-certs\"" Sep 07 04:00:36.877940 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:36.877911 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg"] Sep 07 04:00:36.954220 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:36.954181 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-home\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:36.954420 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:36.954235 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwchd\" (UniqueName: \"kubernetes.io/projected/3ded062d-85fc-4aaa-b20d-54aea673370e-kube-api-access-gwchd\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:36.954420 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:36.954308 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ded062d-85fc-4aaa-b20d-54aea673370e-tls-certs\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:36.954420 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:36.954333 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-tmp-dir\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:36.954420 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:36.954367 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-dshm\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:36.954572 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:36.954476 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-model-cache\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.055248 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.055206 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ded062d-85fc-4aaa-b20d-54aea673370e-tls-certs\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.055248 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.055247 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-tmp-dir\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.055461 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.055277 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-dshm\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.055461 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.055362 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-model-cache\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.055461 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.055403 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-home\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.055461 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.055438 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwchd\" (UniqueName: \"kubernetes.io/projected/3ded062d-85fc-4aaa-b20d-54aea673370e-kube-api-access-gwchd\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.055705 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.055678 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-tmp-dir\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.055784 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.055732 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-model-cache\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.055843 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.055826 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-home\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.057714 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.057689 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-dshm\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.057843 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.057827 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ded062d-85fc-4aaa-b20d-54aea673370e-tls-certs\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.063281 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.063256 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwchd\" (UniqueName: \"kubernetes.io/projected/3ded062d-85fc-4aaa-b20d-54aea673370e-kube-api-access-gwchd\") pod \"e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.180337 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.180245 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:37.462731 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.462410 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg"] Sep 07 04:00:37.465380 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:00:37.465352 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ded062d_85fc_4aaa_b20d_54aea673370e.slice/crio-49ed21518b4a79394a01fee45cb6ef73b8da4b9be25616c18ffed9c14879bf1b WatchSource:0}: Error finding container 49ed21518b4a79394a01fee45cb6ef73b8da4b9be25616c18ffed9c14879bf1b: Status 404 returned error can't find the container with id 49ed21518b4a79394a01fee45cb6ef73b8da4b9be25616c18ffed9c14879bf1b Sep 07 04:00:37.616722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.616678 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" event={"ID":"3ded062d-85fc-4aaa-b20d-54aea673370e","Type":"ContainerStarted","Data":"e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0"} Sep 07 04:00:37.616908 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.616732 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" event={"ID":"3ded062d-85fc-4aaa-b20d-54aea673370e","Type":"ContainerStarted","Data":"49ed21518b4a79394a01fee45cb6ef73b8da4b9be25616c18ffed9c14879bf1b"} Sep 07 04:00:37.637873 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:37.637805 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" podStartSLOduration=1.637785053 podStartE2EDuration="1.637785053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:00:37.6357944 +0000 UTC m=+756.210660691" watchObservedRunningTime="2026-09-07 04:00:37.637785053 +0000 UTC m=+756.212651342" Sep 07 04:00:47.180949 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:47.180907 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:47.181383 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:47.180971 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:47.194266 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:47.194241 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:47.689729 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:47.689702 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:00:49.596754 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:49.596719 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:00:49.691190 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:49.691153 2555 generic.go:356] "Generic (PLEG): container finished" podID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerID="ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e" exitCode=0 Sep 07 04:00:49.691335 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:49.691215 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" event={"ID":"b97b3dc0-690b-4661-86b7-638f1e41cfb2","Type":"ContainerDied","Data":"ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e"} Sep 07 04:00:49.691625 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:49.691600 2555 scope.go:117] "RemoveContainer" containerID="ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e" Sep 07 04:00:49.692912 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:49.692884 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:00:49.692995 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:49.692964 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:00:49.693044 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:49.693003 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:00:50.698133 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:50.698096 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" event={"ID":"b97b3dc0-690b-4661-86b7-638f1e41cfb2","Type":"ContainerStarted","Data":"0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc"} Sep 07 04:00:50.878103 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:50.878074 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:50.878273 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:50.878258 2555 patch_prober.go:28] interesting pod/e2e-autores-908959-openshift-default-f468d74f8-hl98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 07 04:00:50.878351 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:50.878332 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 07 04:00:51.878909 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:51.878877 2555 patch_prober.go:28] interesting pod/e2e-autores-908959-openshift-default-f468d74f8-hl98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 07 04:00:51.879316 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:51.878931 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 07 04:00:52.770775 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:52.770739 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9"] Sep 07 04:00:52.774626 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:52.774608 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" Sep 07 04:00:52.783074 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:52.777349 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 04:00:52.783074 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:52.777500 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs4ff\"" Sep 07 04:00:52.783074 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:52.782482 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9"] Sep 07 04:00:52.878028 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:52.877992 2555 patch_prober.go:28] interesting pod/e2e-autores-908959-openshift-default-f468d74f8-hl98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 07 04:00:52.878196 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:52.878086 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 07 04:00:52.908666 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:52.908621 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b20c58d-4cfc-4187-8261-a9348ee534c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9\" (UID: \"2b20c58d-4cfc-4187-8261-a9348ee534c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" Sep 07 04:00:53.009369 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:53.009329 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b20c58d-4cfc-4187-8261-a9348ee534c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9\" (UID: \"2b20c58d-4cfc-4187-8261-a9348ee534c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" Sep 07 04:00:53.009978 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:53.009951 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b20c58d-4cfc-4187-8261-a9348ee534c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9\" (UID: \"2b20c58d-4cfc-4187-8261-a9348ee534c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" Sep 07 04:00:53.094232 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:53.094140 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" Sep 07 04:00:53.385675 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:53.385618 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9"] Sep 07 04:00:53.388027 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:00:53.388001 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b20c58d_4cfc_4187_8261_a9348ee534c8.slice/crio-519e40a53c2c403af45176abe5cb9bf4ddf023dc56d8f4940f3677ee45e452ab WatchSource:0}: Error finding container 519e40a53c2c403af45176abe5cb9bf4ddf023dc56d8f4940f3677ee45e452ab: Status 404 returned error can't find the container with id 519e40a53c2c403af45176abe5cb9bf4ddf023dc56d8f4940f3677ee45e452ab Sep 07 04:00:53.716422 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:53.716388 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" event={"ID":"2b20c58d-4cfc-4187-8261-a9348ee534c8","Type":"ContainerStarted","Data":"519e40a53c2c403af45176abe5cb9bf4ddf023dc56d8f4940f3677ee45e452ab"} Sep 07 04:00:53.878961 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:53.878917 2555 patch_prober.go:28] interesting pod/e2e-autores-908959-openshift-default-f468d74f8-hl98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 07 04:00:53.879157 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:53.878984 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 07 04:00:54.878988 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:54.878956 2555 patch_prober.go:28] interesting pod/e2e-autores-908959-openshift-default-f468d74f8-hl98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 07 04:00:54.879514 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:54.879009 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 07 04:00:55.878290 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:55.878259 2555 patch_prober.go:28] interesting pod/e2e-autores-908959-openshift-default-f468d74f8-hl98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 07 04:00:55.878479 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:55.878318 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 07 04:00:56.735735 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:56.735699 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" event={"ID":"2b20c58d-4cfc-4187-8261-a9348ee534c8","Type":"ContainerStarted","Data":"58675bd9817179dfa2b0ea820d5420d85a9fd7b66951cee0bd77efeadefe7aae"} Sep 07 04:00:56.750576 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:56.750521 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" podStartSLOduration=1.9849615680000001 podStartE2EDuration="4.750506777s" totalImagesPullingTime="2.765545209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:00:56.749467622 +0000 UTC m=+775.324333912" watchObservedRunningTime="2026-09-07 04:00:56.750506777 +0000 UTC m=+775.325373066" Sep 07 04:00:56.881846 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:56.881822 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:56.882104 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:56.882084 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:56.882788 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:56.882771 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:00:57.742736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:57.742705 2555 generic.go:356] "Generic (PLEG): container finished" podID="2b20c58d-4cfc-4187-8261-a9348ee534c8" containerID="58675bd9817179dfa2b0ea820d5420d85a9fd7b66951cee0bd77efeadefe7aae" exitCode=0 Sep 07 04:00:57.743142 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:57.742777 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" event={"ID":"2b20c58d-4cfc-4187-8261-a9348ee534c8","Type":"ContainerDied","Data":"58675bd9817179dfa2b0ea820d5420d85a9fd7b66951cee0bd77efeadefe7aae"} Sep 07 04:00:58.881727 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:58.881706 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" Sep 07 04:00:58.990676 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:58.990640 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b20c58d-4cfc-4187-8261-a9348ee534c8-maas-api-service-ca\") pod \"2b20c58d-4cfc-4187-8261-a9348ee534c8\" (UID: \"2b20c58d-4cfc-4187-8261-a9348ee534c8\") " Sep 07 04:00:58.991045 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:58.991017 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b20c58d-4cfc-4187-8261-a9348ee534c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b20c58d-4cfc-4187-8261-a9348ee534c8" (UID: "2b20c58d-4cfc-4187-8261-a9348ee534c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:00:59.092369 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:59.092258 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b20c58d-4cfc-4187-8261-a9348ee534c8-maas-api-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:00:59.754179 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:59.754141 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" event={"ID":"2b20c58d-4cfc-4187-8261-a9348ee534c8","Type":"ContainerDied","Data":"519e40a53c2c403af45176abe5cb9bf4ddf023dc56d8f4940f3677ee45e452ab"} Sep 07 04:00:59.754179 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:59.754161 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9" Sep 07 04:00:59.754388 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:00:59.754197 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="519e40a53c2c403af45176abe5cb9bf4ddf023dc56d8f4940f3677ee45e452ab" Sep 07 04:01:06.978742 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:06.978708 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2"] Sep 07 04:01:06.979227 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:06.979114 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b20c58d-4cfc-4187-8261-a9348ee534c8" containerName="revoke-keys" Sep 07 04:01:06.979227 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:06.979128 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b20c58d-4cfc-4187-8261-a9348ee534c8" containerName="revoke-keys" Sep 07 04:01:06.979227 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:06.979210 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b20c58d-4cfc-4187-8261-a9348ee534c8" containerName="revoke-keys" Sep 07 04:01:06.982516 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:06.982499 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:06.984772 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:06.984751 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-9065bcd9-kserve-self-signed-certs\"" Sep 07 04:01:06.990469 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:06.990447 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2"] Sep 07 04:01:07.064856 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.064820 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-home\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.065040 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.064899 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-model-cache\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.065040 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.064922 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tmp-dir\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.065040 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.064939 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-dshm\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.065040 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.064956 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tls-certs\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.065040 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.064995 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxm8\" (UniqueName: \"kubernetes.io/projected/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-kube-api-access-9dxm8\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.165683 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.165652 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-model-cache\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.165683 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.165684 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tmp-dir\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.165920 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.165703 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-dshm\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.165920 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.165723 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tls-certs\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.165920 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.165748 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxm8\" (UniqueName: \"kubernetes.io/projected/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-kube-api-access-9dxm8\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.165920 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.165850 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-home\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.166160 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.166132 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tmp-dir\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.166207 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.166138 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-model-cache\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.166249 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.166222 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-home\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.168034 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.168013 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-dshm\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.168175 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.168158 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tls-certs\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.172895 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.172873 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxm8\" (UniqueName: \"kubernetes.io/projected/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-kube-api-access-9dxm8\") pod \"e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.292945 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.292853 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:07.564319 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.564294 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2"] Sep 07 04:01:07.566373 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:01:07.566345 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa8cdb55_aa6a_483f_aec4_77e0b22fe5f2.slice/crio-0fc20182835dcceef776441e24c1674c13c42de601fdf0cdc1e98ec4abad8709 WatchSource:0}: Error finding container 0fc20182835dcceef776441e24c1674c13c42de601fdf0cdc1e98ec4abad8709: Status 404 returned error can't find the container with id 0fc20182835dcceef776441e24c1674c13c42de601fdf0cdc1e98ec4abad8709 Sep 07 04:01:07.789504 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.789473 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:01:07.805884 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.805850 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" event={"ID":"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2","Type":"ContainerStarted","Data":"89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65"} Sep 07 04:01:07.806025 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.805900 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" event={"ID":"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2","Type":"ContainerStarted","Data":"0fc20182835dcceef776441e24c1674c13c42de601fdf0cdc1e98ec4abad8709"} Sep 07 04:01:07.826919 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:07.826823 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" podStartSLOduration=1.826809793 podStartE2EDuration="1.826809793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:01:07.824765125 +0000 UTC m=+786.399631413" watchObservedRunningTime="2026-09-07 04:01:07.826809793 +0000 UTC m=+786.401676082" Sep 07 04:01:09.617880 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:09.617840 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9"] Sep 07 04:01:09.620334 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:09.620308 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64f80f33-fmgz9"] Sep 07 04:01:09.937920 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:09.937842 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b20c58d-4cfc-4187-8261-a9348ee534c8" path="/var/lib/kubelet/pods/2b20c58d-4cfc-4187-8261-a9348ee534c8/volumes" Sep 07 04:01:10.687338 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.687304 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z"] Sep 07 04:01:10.691232 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.691209 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.693728 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.693706 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e1915828-gw-openshift-default-dockercfg-rxwdz\"" Sep 07 04:01:10.703971 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.703948 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z"] Sep 07 04:01:10.798681 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.798642 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-data\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.798681 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.798683 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f342094c-43e2-401d-aa73-a2283812f587-istiod-ca-cert\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.798931 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.798716 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6htn\" (UniqueName: \"kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-kube-api-access-m6htn\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.798931 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.798741 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-certs\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.798931 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.798779 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-socket\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.798931 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.798848 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f342094c-43e2-401d-aa73-a2283812f587-istio-podinfo\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.798931 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.798886 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-credential-socket\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.798931 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.798916 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-envoy\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.799177 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.799052 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-istio-token\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.900471 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.900431 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-envoy\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.900665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.900498 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-istio-token\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.900665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.900561 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-data\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.900665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.900585 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f342094c-43e2-401d-aa73-a2283812f587-istiod-ca-cert\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.900665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.900613 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6htn\" (UniqueName: \"kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-kube-api-access-m6htn\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.900665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.900638 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-certs\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.900960 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.900675 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-socket\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.900960 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.900723 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f342094c-43e2-401d-aa73-a2283812f587-istio-podinfo\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.900960 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.900767 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-credential-socket\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.901154 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.901083 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-certs\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.901247 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.901224 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-socket\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.901247 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.901241 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-data\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.901354 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.901334 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-credential-socket\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.901489 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.901460 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f342094c-43e2-401d-aa73-a2283812f587-istiod-ca-cert\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.903149 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.903129 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-envoy\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.903322 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.903302 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f342094c-43e2-401d-aa73-a2283812f587-istio-podinfo\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.908835 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.908804 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-istio-token\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:10.909019 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:10.908999 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6htn\" (UniqueName: \"kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-kube-api-access-m6htn\") pod \"e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:11.003537 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:11.003504 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:11.310020 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:11.309996 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z"] Sep 07 04:01:11.312251 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:01:11.312223 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf342094c_43e2_401d_aa73_a2283812f587.slice/crio-a128e3d46fc8dbbadfcf525748dab88e586226625194081e6befc6d3eab4acea WatchSource:0}: Error finding container a128e3d46fc8dbbadfcf525748dab88e586226625194081e6befc6d3eab4acea: Status 404 returned error can't find the container with id a128e3d46fc8dbbadfcf525748dab88e586226625194081e6befc6d3eab4acea Sep 07 04:01:11.314925 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:11.314888 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:01:11.315033 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:11.314957 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:01:11.315033 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:11.314991 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:01:11.829691 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:11.829658 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" event={"ID":"f342094c-43e2-401d-aa73-a2283812f587","Type":"ContainerStarted","Data":"9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af"} Sep 07 04:01:11.830113 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:11.829716 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" event={"ID":"f342094c-43e2-401d-aa73-a2283812f587","Type":"ContainerStarted","Data":"a128e3d46fc8dbbadfcf525748dab88e586226625194081e6befc6d3eab4acea"} Sep 07 04:01:11.849567 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:11.849520 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" podStartSLOduration=1.849503061 podStartE2EDuration="1.849503061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:01:11.847956988 +0000 UTC m=+790.422823278" watchObservedRunningTime="2026-09-07 04:01:11.849503061 +0000 UTC m=+790.424369354" Sep 07 04:01:12.004293 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:12.004257 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:12.009495 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:12.009471 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:12.835838 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:12.835802 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:12.837028 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:12.837002 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:17.293547 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:17.293520 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:17.294003 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:17.293580 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:17.305851 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:17.305826 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:17.874520 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:17.874491 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:28.032381 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:28.031687 2555 patch_prober.go:28] interesting pod/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:35750->10.132.0.51:15021: read: connection reset by peer" start-of-body= Sep 07 04:01:28.032381 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:28.031757 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:35750->10.132.0.51:15021: read: connection reset by peer" Sep 07 04:01:28.924323 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:28.924290 2555 generic.go:356] "Generic (PLEG): container finished" podID="f342094c-43e2-401d-aa73-a2283812f587" containerID="9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af" exitCode=0 Sep 07 04:01:28.924502 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:28.924365 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" event={"ID":"f342094c-43e2-401d-aa73-a2283812f587","Type":"ContainerDied","Data":"9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af"} Sep 07 04:01:28.924804 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:28.924789 2555 scope.go:117] "RemoveContainer" containerID="9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af" Sep 07 04:01:28.925912 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:28.925882 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:01:28.925974 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:28.925959 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:01:28.926011 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:28.925995 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:01:29.459724 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:29.459694 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2"] Sep 07 04:01:29.938832 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:29.938727 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" event={"ID":"f342094c-43e2-401d-aa73-a2283812f587","Type":"ContainerStarted","Data":"75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e"} Sep 07 04:01:30.004447 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:30.004407 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:30.004638 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:30.004617 2555 patch_prober.go:28] interesting pod/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 07 04:01:30.004684 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:30.004665 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 07 04:01:31.004625 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:31.004588 2555 patch_prober.go:28] interesting pod/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 07 04:01:31.005023 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:31.004644 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 07 04:01:32.004161 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:32.004124 2555 patch_prober.go:28] interesting pod/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 07 04:01:32.004352 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:32.004188 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 07 04:01:32.007383 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:32.007355 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:01:33.004168 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:33.004125 2555 patch_prober.go:28] interesting pod/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 07 04:01:33.004390 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:33.004183 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 07 04:01:34.004496 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:34.004458 2555 patch_prober.go:28] interesting pod/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 07 04:01:34.004891 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:34.004524 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 07 04:01:35.004325 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:35.004275 2555 patch_prober.go:28] interesting pod/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 07 04:01:35.004550 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:35.004361 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 07 04:01:36.008193 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:36.008162 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:36.008569 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:36.008539 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:36.009148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:36.009130 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:38.194971 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:38.194932 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd"] Sep 07 04:01:38.200001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:38.199978 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd" Sep 07 04:01:38.204351 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:38.203876 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 04:01:38.204351 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:38.203876 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs4ff\"" Sep 07 04:01:38.205658 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:38.205639 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd"] Sep 07 04:01:38.281871 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:38.281833 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55420757-83d2-4d6a-b939-65394d0b523a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd\" (UID: \"55420757-83d2-4d6a-b939-65394d0b523a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd" Sep 07 04:01:38.382842 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:38.382803 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55420757-83d2-4d6a-b939-65394d0b523a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd\" (UID: \"55420757-83d2-4d6a-b939-65394d0b523a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd" Sep 07 04:01:38.383430 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:38.383411 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55420757-83d2-4d6a-b939-65394d0b523a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd\" (UID: \"55420757-83d2-4d6a-b939-65394d0b523a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd" Sep 07 04:01:38.512028 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:38.511997 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd" Sep 07 04:01:38.782190 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:38.782165 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd"] Sep 07 04:01:38.784438 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:01:38.784408 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55420757_83d2_4d6a_b939_65394d0b523a.slice/crio-4caa5169ad0f485c519d8e0c0f8cfb971e175fef2b85ef87dc2c317d90ce0b9b WatchSource:0}: Error finding container 4caa5169ad0f485c519d8e0c0f8cfb971e175fef2b85ef87dc2c317d90ce0b9b: Status 404 returned error can't find the container with id 4caa5169ad0f485c519d8e0c0f8cfb971e175fef2b85ef87dc2c317d90ce0b9b Sep 07 04:01:39.000124 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:39.000088 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd" event={"ID":"55420757-83d2-4d6a-b939-65394d0b523a","Type":"ContainerStarted","Data":"4caa5169ad0f485c519d8e0c0f8cfb971e175fef2b85ef87dc2c317d90ce0b9b"} Sep 07 04:01:40.010743 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:40.010707 2555 generic.go:356] "Generic (PLEG): container finished" podID="55420757-83d2-4d6a-b939-65394d0b523a" containerID="21729231c8582397de6981ea55c315eb040b4022e1f56610a0a499c06a7b5a0d" exitCode=0 Sep 07 04:01:40.011152 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:40.010798 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd" event={"ID":"55420757-83d2-4d6a-b939-65394d0b523a","Type":"ContainerDied","Data":"21729231c8582397de6981ea55c315eb040b4022e1f56610a0a499c06a7b5a0d"} Sep 07 04:01:41.154563 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:41.154534 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd" Sep 07 04:01:41.203774 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:41.203741 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55420757-83d2-4d6a-b939-65394d0b523a-maas-api-service-ca\") pod \"55420757-83d2-4d6a-b939-65394d0b523a\" (UID: \"55420757-83d2-4d6a-b939-65394d0b523a\") " Sep 07 04:01:41.204118 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:41.204096 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55420757-83d2-4d6a-b939-65394d0b523a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55420757-83d2-4d6a-b939-65394d0b523a" (UID: "55420757-83d2-4d6a-b939-65394d0b523a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:01:41.305101 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:41.305008 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55420757-83d2-4d6a-b939-65394d0b523a-maas-api-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:42.021809 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:42.021780 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd" Sep 07 04:01:42.021809 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:42.021788 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd" event={"ID":"55420757-83d2-4d6a-b939-65394d0b523a","Type":"ContainerDied","Data":"4caa5169ad0f485c519d8e0c0f8cfb971e175fef2b85ef87dc2c317d90ce0b9b"} Sep 07 04:01:42.022001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:42.021838 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4caa5169ad0f485c519d8e0c0f8cfb971e175fef2b85ef87dc2c317d90ce0b9b" Sep 07 04:01:44.479689 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.479645 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" podUID="fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" containerName="main" containerID="cri-o://89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65" gracePeriod=45 Sep 07 04:01:44.833644 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.833363 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:44.948862 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.948832 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tls-certs\") pod \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " Sep 07 04:01:44.948862 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.948870 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-dshm\") pod \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " Sep 07 04:01:44.949217 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.948935 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-home\") pod \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " Sep 07 04:01:44.949217 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.948973 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-model-cache\") pod \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " Sep 07 04:01:44.949217 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.949021 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tmp-dir\") pod \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " Sep 07 04:01:44.949217 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.949109 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dxm8\" (UniqueName: \"kubernetes.io/projected/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-kube-api-access-9dxm8\") pod \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\" (UID: \"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2\") " Sep 07 04:01:44.949434 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.949393 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" (UID: "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:01:44.949512 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.949431 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" (UID: "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:01:44.949512 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.949495 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-home" (OuterVolumeSpecName: "home") pod "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" (UID: "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:01:44.951600 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.951449 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" (UID: "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:01:44.951726 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.951632 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-kube-api-access-9dxm8" (OuterVolumeSpecName: "kube-api-access-9dxm8") pod "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" (UID: "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2"). InnerVolumeSpecName "kube-api-access-9dxm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:01:44.951909 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.951883 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-dshm" (OuterVolumeSpecName: "dshm") pod "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" (UID: "fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:01:44.999610 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:44.999577 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd"] Sep 07 04:01:45.004481 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.004454 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1915828-6jrdd"] Sep 07 04:01:45.040274 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.040180 2555 generic.go:356] "Generic (PLEG): container finished" podID="fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" containerID="89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65" exitCode=0 Sep 07 04:01:45.040274 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.040250 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" Sep 07 04:01:45.040463 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.040271 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" event={"ID":"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2","Type":"ContainerDied","Data":"89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65"} Sep 07 04:01:45.040463 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.040321 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2" event={"ID":"fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2","Type":"ContainerDied","Data":"0fc20182835dcceef776441e24c1674c13c42de601fdf0cdc1e98ec4abad8709"} Sep 07 04:01:45.040463 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.040342 2555 scope.go:117] "RemoveContainer" containerID="89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65" Sep 07 04:01:45.050108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.050082 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tmp-dir\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:45.050234 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.050110 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dxm8\" (UniqueName: \"kubernetes.io/projected/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-kube-api-access-9dxm8\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:45.050234 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.050125 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-tls-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:45.050234 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.050140 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-dshm\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:45.050234 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.050152 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-home\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:45.050234 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.050165 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2-model-cache\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:45.059029 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.059004 2555 scope.go:117] "RemoveContainer" containerID="89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65" Sep 07 04:01:45.059388 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:01:45.059366 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65\": container with ID starting with 89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65 not found: ID does not exist" containerID="89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65" Sep 07 04:01:45.059463 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.059401 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65"} err="failed to get container status \"89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65\": rpc error: code = NotFound desc = could not find container \"89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65\": container with ID starting with 89587eceab65de92a41c79511569c4695b687ab9df39ca5a0b86c0e29e505a65 not found: ID does not exist" Sep 07 04:01:45.064383 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.064353 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2"] Sep 07 04:01:45.068587 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.068567 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-9065bcd9-kserve-77854c496c-42bb2"] Sep 07 04:01:45.140042 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.140009 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z"] Sep 07 04:01:45.140344 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.140312 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" containerID="cri-o://75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e" gracePeriod=30 Sep 07 04:01:45.938190 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.938158 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55420757-83d2-4d6a-b939-65394d0b523a" path="/var/lib/kubelet/pods/55420757-83d2-4d6a-b939-65394d0b523a/volumes" Sep 07 04:01:45.938695 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:45.938678 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" path="/var/lib/kubelet/pods/fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2/volumes" Sep 07 04:01:50.003587 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.003553 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv"] Sep 07 04:01:50.004868 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.004134 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55420757-83d2-4d6a-b939-65394d0b523a" containerName="revoke-keys" Sep 07 04:01:50.004868 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.004158 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="55420757-83d2-4d6a-b939-65394d0b523a" containerName="revoke-keys" Sep 07 04:01:50.004868 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.004248 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" containerName="main" Sep 07 04:01:50.004868 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.004261 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" containerName="main" Sep 07 04:01:50.005200 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.004937 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="55420757-83d2-4d6a-b939-65394d0b523a" containerName="revoke-keys" Sep 07 04:01:50.005200 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.004959 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8cdb55-aa6a-483f-aec4-77e0b22fe5f2" containerName="main" Sep 07 04:01:50.009514 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.009496 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.012144 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.012120 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 07 04:01:50.013243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.013156 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4d42cf5d-serving-cert\"" Sep 07 04:01:50.013243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.013218 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-m9mvz\"" Sep 07 04:01:50.013243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.013229 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 07 04:01:50.013243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.013248 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 04:01:50.017496 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.017462 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv"] Sep 07 04:01:50.098439 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.098403 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-maas-api-tls\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.098619 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.098449 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a248f122-e72e-469a-9246-c1eb7fcee008-openshift-service-ca\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.098619 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.098568 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pspjv\" (UniqueName: \"kubernetes.io/projected/a248f122-e72e-469a-9246-c1eb7fcee008-kube-api-access-pspjv\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.098619 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.098605 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-metrics-certs\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.199657 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.199623 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a248f122-e72e-469a-9246-c1eb7fcee008-openshift-service-ca\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.199829 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.199802 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pspjv\" (UniqueName: \"kubernetes.io/projected/a248f122-e72e-469a-9246-c1eb7fcee008-kube-api-access-pspjv\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.199882 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.199864 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-metrics-certs\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.199977 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.199961 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-maas-api-tls\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.200426 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.200402 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a248f122-e72e-469a-9246-c1eb7fcee008-openshift-service-ca\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.202209 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.202186 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-metrics-certs\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.202307 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.202247 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-maas-api-tls\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.207157 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.207139 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pspjv\" (UniqueName: \"kubernetes.io/projected/a248f122-e72e-469a-9246-c1eb7fcee008-kube-api-access-pspjv\") pod \"maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.323573 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.323542 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:50.398766 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.398740 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:50.502427 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502377 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-credential-socket\") pod \"f342094c-43e2-401d-aa73-a2283812f587\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " Sep 07 04:01:50.502596 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502459 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-envoy\") pod \"f342094c-43e2-401d-aa73-a2283812f587\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " Sep 07 04:01:50.502596 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502497 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-data\") pod \"f342094c-43e2-401d-aa73-a2283812f587\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " Sep 07 04:01:50.502596 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502540 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f342094c-43e2-401d-aa73-a2283812f587-istio-podinfo\") pod \"f342094c-43e2-401d-aa73-a2283812f587\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " Sep 07 04:01:50.502596 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502578 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f342094c-43e2-401d-aa73-a2283812f587-istiod-ca-cert\") pod \"f342094c-43e2-401d-aa73-a2283812f587\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " Sep 07 04:01:50.502828 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502660 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-certs\") pod \"f342094c-43e2-401d-aa73-a2283812f587\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " Sep 07 04:01:50.502828 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502685 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-istio-token\") pod \"f342094c-43e2-401d-aa73-a2283812f587\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " Sep 07 04:01:50.502828 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502715 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f342094c-43e2-401d-aa73-a2283812f587" (UID: "f342094c-43e2-401d-aa73-a2283812f587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:01:50.502828 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502744 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-socket\") pod \"f342094c-43e2-401d-aa73-a2283812f587\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " Sep 07 04:01:50.503033 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502970 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-data" (OuterVolumeSpecName: "istio-data") pod "f342094c-43e2-401d-aa73-a2283812f587" (UID: "f342094c-43e2-401d-aa73-a2283812f587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:01:50.503033 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.502984 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f342094c-43e2-401d-aa73-a2283812f587" (UID: "f342094c-43e2-401d-aa73-a2283812f587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:01:50.503148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.503127 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6htn\" (UniqueName: \"kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-kube-api-access-m6htn\") pod \"f342094c-43e2-401d-aa73-a2283812f587\" (UID: \"f342094c-43e2-401d-aa73-a2283812f587\") " Sep 07 04:01:50.503201 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.503151 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f342094c-43e2-401d-aa73-a2283812f587-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f342094c-43e2-401d-aa73-a2283812f587" (UID: "f342094c-43e2-401d-aa73-a2283812f587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:01:50.503567 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.503494 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f342094c-43e2-401d-aa73-a2283812f587-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:50.503567 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.503516 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:50.503567 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.503530 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:50.503567 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.503544 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:50.503923 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.503893 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f342094c-43e2-401d-aa73-a2283812f587" (UID: "f342094c-43e2-401d-aa73-a2283812f587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:01:50.505742 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.505715 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-istio-token" (OuterVolumeSpecName: "istio-token") pod "f342094c-43e2-401d-aa73-a2283812f587" (UID: "f342094c-43e2-401d-aa73-a2283812f587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:01:50.505843 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.505777 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f342094c-43e2-401d-aa73-a2283812f587" (UID: "f342094c-43e2-401d-aa73-a2283812f587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:01:50.505902 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.505852 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f342094c-43e2-401d-aa73-a2283812f587-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f342094c-43e2-401d-aa73-a2283812f587" (UID: "f342094c-43e2-401d-aa73-a2283812f587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:01:50.505989 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.505967 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-kube-api-access-m6htn" (OuterVolumeSpecName: "kube-api-access-m6htn") pod "f342094c-43e2-401d-aa73-a2283812f587" (UID: "f342094c-43e2-401d-aa73-a2283812f587"). InnerVolumeSpecName "kube-api-access-m6htn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:01:50.600957 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.600932 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv"] Sep 07 04:01:50.603110 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:01:50.603081 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda248f122_e72e_469a_9246_c1eb7fcee008.slice/crio-6ea2a2bd969a1eb494ae6aaea48ded636e54583627c80144647e095afdf24ef8 WatchSource:0}: Error finding container 6ea2a2bd969a1eb494ae6aaea48ded636e54583627c80144647e095afdf24ef8: Status 404 returned error can't find the container with id 6ea2a2bd969a1eb494ae6aaea48ded636e54583627c80144647e095afdf24ef8 Sep 07 04:01:50.604040 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.604019 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6htn\" (UniqueName: \"kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-kube-api-access-m6htn\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:50.604040 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.604041 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:50.604200 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.604051 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f342094c-43e2-401d-aa73-a2283812f587-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:50.604200 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.604085 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f342094c-43e2-401d-aa73-a2283812f587-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:50.604200 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:50.604094 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f342094c-43e2-401d-aa73-a2283812f587-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:01:51.074893 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.074855 2555 generic.go:356] "Generic (PLEG): container finished" podID="f342094c-43e2-401d-aa73-a2283812f587" containerID="75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e" exitCode=0 Sep 07 04:01:51.075325 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.074927 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" Sep 07 04:01:51.075325 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.074939 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" event={"ID":"f342094c-43e2-401d-aa73-a2283812f587","Type":"ContainerDied","Data":"75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e"} Sep 07 04:01:51.075325 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.074995 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z" event={"ID":"f342094c-43e2-401d-aa73-a2283812f587","Type":"ContainerDied","Data":"a128e3d46fc8dbbadfcf525748dab88e586226625194081e6befc6d3eab4acea"} Sep 07 04:01:51.075325 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.075018 2555 scope.go:117] "RemoveContainer" containerID="75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e" Sep 07 04:01:51.076539 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.076504 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" event={"ID":"a248f122-e72e-469a-9246-c1eb7fcee008","Type":"ContainerStarted","Data":"6ea2a2bd969a1eb494ae6aaea48ded636e54583627c80144647e095afdf24ef8"} Sep 07 04:01:51.084302 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.084288 2555 scope.go:117] "RemoveContainer" containerID="9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af" Sep 07 04:01:51.092445 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.092427 2555 scope.go:117] "RemoveContainer" containerID="75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e" Sep 07 04:01:51.092695 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:01:51.092677 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e\": container with ID starting with 75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e not found: ID does not exist" containerID="75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e" Sep 07 04:01:51.092742 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.092703 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e"} err="failed to get container status \"75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e\": rpc error: code = NotFound desc = could not find container \"75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e\": container with ID starting with 75c2143b45f65545dd85faa1a4ba910ba897d7563afbb284aedf511e0436af5e not found: ID does not exist" Sep 07 04:01:51.092742 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.092719 2555 scope.go:117] "RemoveContainer" containerID="9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af" Sep 07 04:01:51.092950 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:01:51.092932 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af\": container with ID starting with 9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af not found: ID does not exist" containerID="9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af" Sep 07 04:01:51.092992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.092955 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af"} err="failed to get container status \"9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af\": rpc error: code = NotFound desc = could not find container \"9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af\": container with ID starting with 9f0d5411d6bfec7885c4c720bf466bf9cca4b203d19d9aae837adba4f033a5af not found: ID does not exist" Sep 07 04:01:51.095884 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.095864 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z"] Sep 07 04:01:51.101729 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.101707 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e1915828-gw-openshift-default-b4784c8d8-hnw4z"] Sep 07 04:01:51.501691 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.501654 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:01:51.939955 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:51.939875 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f342094c-43e2-401d-aa73-a2283812f587" path="/var/lib/kubelet/pods/f342094c-43e2-401d-aa73-a2283812f587/volumes" Sep 07 04:01:53.094967 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:53.094928 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" event={"ID":"a248f122-e72e-469a-9246-c1eb7fcee008","Type":"ContainerStarted","Data":"74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0"} Sep 07 04:01:53.095515 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:53.095088 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:01:53.114280 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:53.114220 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" podStartSLOduration=2.542644838 podStartE2EDuration="4.114202713s" totalImagesPullingTime="1.571557875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:01:53.112321405 +0000 UTC m=+831.687187694" watchObservedRunningTime="2026-09-07 04:01:53.114202713 +0000 UTC m=+831.689069002" Sep 07 04:01:56.855786 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:56.855748 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg"] Sep 07 04:01:59.105132 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:01:59.105103 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:02:08.489821 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:08.489779 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:02:09.708182 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:09.708142 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x"] Sep 07 04:02:09.708766 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:09.708424 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" containerID="cri-o://0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc" gracePeriod=30 Sep 07 04:02:10.421461 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.421414 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh"] Sep 07 04:02:10.422067 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.422037 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" Sep 07 04:02:10.422147 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.422075 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" Sep 07 04:02:10.422239 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.422226 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" Sep 07 04:02:10.422296 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.422245 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" Sep 07 04:02:10.427450 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.427427 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.430309 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.430289 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-a84f90-openshift-default-dockercfg-bcq97\"" Sep 07 04:02:10.439008 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.438981 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh"] Sep 07 04:02:10.604007 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.603968 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-socket\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.604211 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.604027 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkdc\" (UniqueName: \"kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-kube-api-access-xzkdc\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.604211 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.604100 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50d648ab-ba1e-4c70-98d4-543361d12798-istio-podinfo\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.604211 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.604131 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-envoy\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.604211 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.604175 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-istio-token\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.604380 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.604223 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-certs\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.604380 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.604271 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-data\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.604380 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.604315 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50d648ab-ba1e-4c70-98d4-543361d12798-istiod-ca-cert\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.604380 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.604342 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-credential-socket\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705256 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705219 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-socket\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705455 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705267 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkdc\" (UniqueName: \"kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-kube-api-access-xzkdc\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705455 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705290 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50d648ab-ba1e-4c70-98d4-543361d12798-istio-podinfo\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705455 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705310 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-envoy\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705455 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705340 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-istio-token\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705455 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705374 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-certs\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705728 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705535 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-data\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705728 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705606 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50d648ab-ba1e-4c70-98d4-543361d12798-istiod-ca-cert\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705728 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705640 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-credential-socket\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705888 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705725 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-socket\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705888 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705745 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-certs\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.705888 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705793 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-data\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.706046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.705943 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-credential-socket\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.706366 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.706346 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50d648ab-ba1e-4c70-98d4-543361d12798-istiod-ca-cert\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.707560 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.707540 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-envoy\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.707937 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.707920 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50d648ab-ba1e-4c70-98d4-543361d12798-istio-podinfo\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.712886 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.712864 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-istio-token\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.713213 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.712959 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkdc\" (UniqueName: \"kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-kube-api-access-xzkdc\") pod \"e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:10.742140 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:10.742110 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:11.025833 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.025809 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh"] Sep 07 04:02:11.028376 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:02:11.028349 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d648ab_ba1e_4c70_98d4_543361d12798.slice/crio-cd84f20bc38bc8b0f086cff83c171407b90b4c90904f529bbb41b7d719bd09a1 WatchSource:0}: Error finding container cd84f20bc38bc8b0f086cff83c171407b90b4c90904f529bbb41b7d719bd09a1: Status 404 returned error can't find the container with id cd84f20bc38bc8b0f086cff83c171407b90b4c90904f529bbb41b7d719bd09a1 Sep 07 04:02:11.032872 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.031250 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:11.032872 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.031407 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:11.032872 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.031499 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:11.189832 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.189796 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" event={"ID":"50d648ab-ba1e-4c70-98d4-543361d12798","Type":"ContainerStarted","Data":"49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf"} Sep 07 04:02:11.189992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.189896 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" event={"ID":"50d648ab-ba1e-4c70-98d4-543361d12798","Type":"ContainerStarted","Data":"cd84f20bc38bc8b0f086cff83c171407b90b4c90904f529bbb41b7d719bd09a1"} Sep 07 04:02:11.212406 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.212345 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" podStartSLOduration=1.212323942 podStartE2EDuration="1.212323942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:02:11.211266342 +0000 UTC m=+849.786132630" watchObservedRunningTime="2026-09-07 04:02:11.212323942 +0000 UTC m=+849.787190235" Sep 07 04:02:11.742629 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.742594 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:11.876461 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.876409 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" podUID="3ded062d-85fc-4aaa-b20d-54aea673370e" containerName="main" containerID="cri-o://e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0" gracePeriod=45 Sep 07 04:02:11.885112 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.885080 2555 patch_prober.go:28] interesting pod/e2e-autores-908959-openshift-default-f468d74f8-hl98x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 04:02:11.885210 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:11.885157 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 04:02:12.140374 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.140347 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:02:12.196283 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.196250 2555 generic.go:356] "Generic (PLEG): container finished" podID="3ded062d-85fc-4aaa-b20d-54aea673370e" containerID="e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0" exitCode=0 Sep 07 04:02:12.196447 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.196286 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" Sep 07 04:02:12.196447 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.196329 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" event={"ID":"3ded062d-85fc-4aaa-b20d-54aea673370e","Type":"ContainerDied","Data":"e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0"} Sep 07 04:02:12.196447 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.196369 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg" event={"ID":"3ded062d-85fc-4aaa-b20d-54aea673370e","Type":"ContainerDied","Data":"49ed21518b4a79394a01fee45cb6ef73b8da4b9be25616c18ffed9c14879bf1b"} Sep 07 04:02:12.196447 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.196391 2555 scope.go:117] "RemoveContainer" containerID="e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0" Sep 07 04:02:12.205583 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.205564 2555 scope.go:117] "RemoveContainer" containerID="e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0" Sep 07 04:02:12.205858 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:02:12.205840 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0\": container with ID starting with e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0 not found: ID does not exist" containerID="e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0" Sep 07 04:02:12.205927 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.205868 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0"} err="failed to get container status \"e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0\": rpc error: code = NotFound desc = could not find container \"e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0\": container with ID starting with e91932dadc5d53fd7f0ba7951a066bb63807919761126fc853039365d808b3b0 not found: ID does not exist" Sep 07 04:02:12.322950 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.322866 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-model-cache\") pod \"3ded062d-85fc-4aaa-b20d-54aea673370e\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " Sep 07 04:02:12.322950 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.322939 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-dshm\") pod \"3ded062d-85fc-4aaa-b20d-54aea673370e\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " Sep 07 04:02:12.323202 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.322966 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ded062d-85fc-4aaa-b20d-54aea673370e-tls-certs\") pod \"3ded062d-85fc-4aaa-b20d-54aea673370e\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " Sep 07 04:02:12.323202 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.323019 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-home\") pod \"3ded062d-85fc-4aaa-b20d-54aea673370e\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " Sep 07 04:02:12.323202 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.323115 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwchd\" (UniqueName: \"kubernetes.io/projected/3ded062d-85fc-4aaa-b20d-54aea673370e-kube-api-access-gwchd\") pod \"3ded062d-85fc-4aaa-b20d-54aea673370e\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " Sep 07 04:02:12.323202 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.323140 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ded062d-85fc-4aaa-b20d-54aea673370e" (UID: "3ded062d-85fc-4aaa-b20d-54aea673370e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:02:12.323202 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.323160 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-tmp-dir\") pod \"3ded062d-85fc-4aaa-b20d-54aea673370e\" (UID: \"3ded062d-85fc-4aaa-b20d-54aea673370e\") " Sep 07 04:02:12.323458 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.323314 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-home" (OuterVolumeSpecName: "home") pod "3ded062d-85fc-4aaa-b20d-54aea673370e" (UID: "3ded062d-85fc-4aaa-b20d-54aea673370e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:02:12.323458 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.323421 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ded062d-85fc-4aaa-b20d-54aea673370e" (UID: "3ded062d-85fc-4aaa-b20d-54aea673370e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:02:12.323624 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.323603 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-tmp-dir\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:12.323686 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.323627 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-model-cache\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:12.323686 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.323640 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-home\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:12.325205 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.325179 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ded062d-85fc-4aaa-b20d-54aea673370e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ded062d-85fc-4aaa-b20d-54aea673370e" (UID: "3ded062d-85fc-4aaa-b20d-54aea673370e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:02:12.325329 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.325251 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ded062d-85fc-4aaa-b20d-54aea673370e-kube-api-access-gwchd" (OuterVolumeSpecName: "kube-api-access-gwchd") pod "3ded062d-85fc-4aaa-b20d-54aea673370e" (UID: "3ded062d-85fc-4aaa-b20d-54aea673370e"). InnerVolumeSpecName "kube-api-access-gwchd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:02:12.325379 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.325333 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-dshm" (OuterVolumeSpecName: "dshm") pod "3ded062d-85fc-4aaa-b20d-54aea673370e" (UID: "3ded062d-85fc-4aaa-b20d-54aea673370e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:02:12.425071 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.425024 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ded062d-85fc-4aaa-b20d-54aea673370e-tls-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:12.425236 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.425095 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwchd\" (UniqueName: \"kubernetes.io/projected/3ded062d-85fc-4aaa-b20d-54aea673370e-kube-api-access-gwchd\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:12.425236 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.425105 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ded062d-85fc-4aaa-b20d-54aea673370e-dshm\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:12.517032 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.517003 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg"] Sep 07 04:02:12.521042 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.521013 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-9065bcd9-kserve-7cccbb7845-dlgfg"] Sep 07 04:02:12.748206 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:12.748178 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:13.202789 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:13.202761 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:13.203759 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:13.203737 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:13.936514 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:13.936471 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ded062d-85fc-4aaa-b20d-54aea673370e" path="/var/lib/kubelet/pods/3ded062d-85fc-4aaa-b20d-54aea673370e/volumes" Sep 07 04:02:14.983281 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:14.983220 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:02:15.051452 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051418 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-socket\") pod \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " Sep 07 04:02:15.051452 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051459 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-data\") pod \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " Sep 07 04:02:15.051706 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051497 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-certs\") pod \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " Sep 07 04:02:15.051706 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051525 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-token\") pod \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " Sep 07 04:02:15.051706 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051610 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wzg8\" (UniqueName: \"kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-kube-api-access-5wzg8\") pod \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " Sep 07 04:02:15.051706 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051638 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-podinfo\") pod \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " Sep 07 04:02:15.051706 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051696 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istiod-ca-cert\") pod \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " Sep 07 04:02:15.051962 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051722 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-credential-socket\") pod \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " Sep 07 04:02:15.051962 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051758 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-envoy\") pod \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\" (UID: \"b97b3dc0-690b-4661-86b7-638f1e41cfb2\") " Sep 07 04:02:15.051962 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051748 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-data" (OuterVolumeSpecName: "istio-data") pod "b97b3dc0-690b-4661-86b7-638f1e41cfb2" (UID: "b97b3dc0-690b-4661-86b7-638f1e41cfb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:02:15.051962 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.051823 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b97b3dc0-690b-4661-86b7-638f1e41cfb2" (UID: "b97b3dc0-690b-4661-86b7-638f1e41cfb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:02:15.052203 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.052010 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b97b3dc0-690b-4661-86b7-638f1e41cfb2" (UID: "b97b3dc0-690b-4661-86b7-638f1e41cfb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:02:15.052203 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.052079 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b97b3dc0-690b-4661-86b7-638f1e41cfb2" (UID: "b97b3dc0-690b-4661-86b7-638f1e41cfb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:02:15.052203 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.052135 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b97b3dc0-690b-4661-86b7-638f1e41cfb2" (UID: "b97b3dc0-690b-4661-86b7-638f1e41cfb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:02:15.052203 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.052156 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:15.052203 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.052173 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:15.052203 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.052183 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:15.052203 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.052191 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:15.053870 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.053847 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b97b3dc0-690b-4661-86b7-638f1e41cfb2" (UID: "b97b3dc0-690b-4661-86b7-638f1e41cfb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:02:15.053983 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.053938 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-kube-api-access-5wzg8" (OuterVolumeSpecName: "kube-api-access-5wzg8") pod "b97b3dc0-690b-4661-86b7-638f1e41cfb2" (UID: "b97b3dc0-690b-4661-86b7-638f1e41cfb2"). InnerVolumeSpecName "kube-api-access-5wzg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:02:15.054237 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.054218 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b97b3dc0-690b-4661-86b7-638f1e41cfb2" (UID: "b97b3dc0-690b-4661-86b7-638f1e41cfb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:02:15.054304 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.054230 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-token" (OuterVolumeSpecName: "istio-token") pod "b97b3dc0-690b-4661-86b7-638f1e41cfb2" (UID: "b97b3dc0-690b-4661-86b7-638f1e41cfb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:02:15.152646 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.152612 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:15.152646 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.152641 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97b3dc0-690b-4661-86b7-638f1e41cfb2-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:15.152646 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.152651 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:15.152853 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.152660 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzg8\" (UniqueName: \"kubernetes.io/projected/b97b3dc0-690b-4661-86b7-638f1e41cfb2-kube-api-access-5wzg8\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:15.152853 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.152669 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97b3dc0-690b-4661-86b7-638f1e41cfb2-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:15.215202 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.215169 2555 generic.go:356] "Generic (PLEG): container finished" podID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerID="0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc" exitCode=0 Sep 07 04:02:15.215344 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.215221 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" Sep 07 04:02:15.215344 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.215256 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" event={"ID":"b97b3dc0-690b-4661-86b7-638f1e41cfb2","Type":"ContainerDied","Data":"0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc"} Sep 07 04:02:15.215344 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.215315 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x" event={"ID":"b97b3dc0-690b-4661-86b7-638f1e41cfb2","Type":"ContainerDied","Data":"fdbdf265df846f2a23df660b804721c115d23cad707d6f9be9de4ca72e606dc7"} Sep 07 04:02:15.215468 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.215346 2555 scope.go:117] "RemoveContainer" containerID="0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc" Sep 07 04:02:15.224760 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.224744 2555 scope.go:117] "RemoveContainer" containerID="ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e" Sep 07 04:02:15.233194 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.233174 2555 scope.go:117] "RemoveContainer" containerID="0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc" Sep 07 04:02:15.233534 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:02:15.233488 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc\": container with ID starting with 0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc not found: ID does not exist" containerID="0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc" Sep 07 04:02:15.233585 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.233523 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc"} err="failed to get container status \"0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc\": rpc error: code = NotFound desc = could not find container \"0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc\": container with ID starting with 0c328a427dfffbdd329e001ba37cad40d0b1cf79d70b7d8f63184ffbfd6aafbc not found: ID does not exist" Sep 07 04:02:15.233585 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.233547 2555 scope.go:117] "RemoveContainer" containerID="ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e" Sep 07 04:02:15.233829 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:02:15.233810 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e\": container with ID starting with ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e not found: ID does not exist" containerID="ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e" Sep 07 04:02:15.233883 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.233834 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e"} err="failed to get container status \"ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e\": rpc error: code = NotFound desc = could not find container \"ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e\": container with ID starting with ebd63f5c49c18d385d43346dc2a6649beaf795ea8696f3033679530207e3090e not found: ID does not exist" Sep 07 04:02:15.236527 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.236505 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x"] Sep 07 04:02:15.240346 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.240329 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-908959-openshift-default-f468d74f8-hl98x"] Sep 07 04:02:15.941683 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:15.941647 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" path="/var/lib/kubelet/pods/b97b3dc0-690b-4661-86b7-638f1e41cfb2/volumes" Sep 07 04:02:20.664103 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:20.664045 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv"] Sep 07 04:02:20.664601 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:20.664324 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" podUID="a248f122-e72e-469a-9246-c1eb7fcee008" containerName="maas-api" containerID="cri-o://74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0" gracePeriod=30 Sep 07 04:02:20.943317 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:20.943028 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:02:21.009211 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.009181 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-maas-api-tls\") pod \"a248f122-e72e-469a-9246-c1eb7fcee008\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " Sep 07 04:02:21.009386 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.009242 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-metrics-certs\") pod \"a248f122-e72e-469a-9246-c1eb7fcee008\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " Sep 07 04:02:21.009386 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.009266 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pspjv\" (UniqueName: \"kubernetes.io/projected/a248f122-e72e-469a-9246-c1eb7fcee008-kube-api-access-pspjv\") pod \"a248f122-e72e-469a-9246-c1eb7fcee008\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " Sep 07 04:02:21.009386 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.009345 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a248f122-e72e-469a-9246-c1eb7fcee008-openshift-service-ca\") pod \"a248f122-e72e-469a-9246-c1eb7fcee008\" (UID: \"a248f122-e72e-469a-9246-c1eb7fcee008\") " Sep 07 04:02:21.009873 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.009842 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a248f122-e72e-469a-9246-c1eb7fcee008-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a248f122-e72e-469a-9246-c1eb7fcee008" (UID: "a248f122-e72e-469a-9246-c1eb7fcee008"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:02:21.011830 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.011762 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a248f122-e72e-469a-9246-c1eb7fcee008-kube-api-access-pspjv" (OuterVolumeSpecName: "kube-api-access-pspjv") pod "a248f122-e72e-469a-9246-c1eb7fcee008" (UID: "a248f122-e72e-469a-9246-c1eb7fcee008"). InnerVolumeSpecName "kube-api-access-pspjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:02:21.011935 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.011851 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a248f122-e72e-469a-9246-c1eb7fcee008" (UID: "a248f122-e72e-469a-9246-c1eb7fcee008"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:02:21.011935 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.011878 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a248f122-e72e-469a-9246-c1eb7fcee008" (UID: "a248f122-e72e-469a-9246-c1eb7fcee008"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:02:21.110730 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.110695 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-metrics-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:21.110730 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.110725 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pspjv\" (UniqueName: \"kubernetes.io/projected/a248f122-e72e-469a-9246-c1eb7fcee008-kube-api-access-pspjv\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:21.110730 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.110735 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a248f122-e72e-469a-9246-c1eb7fcee008-openshift-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:21.110968 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.110744 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a248f122-e72e-469a-9246-c1eb7fcee008-maas-api-tls\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:21.258511 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.258475 2555 generic.go:356] "Generic (PLEG): container finished" podID="a248f122-e72e-469a-9246-c1eb7fcee008" containerID="74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0" exitCode=0 Sep 07 04:02:21.258723 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.258568 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" event={"ID":"a248f122-e72e-469a-9246-c1eb7fcee008","Type":"ContainerDied","Data":"74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0"} Sep 07 04:02:21.258723 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.258620 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" event={"ID":"a248f122-e72e-469a-9246-c1eb7fcee008","Type":"ContainerDied","Data":"6ea2a2bd969a1eb494ae6aaea48ded636e54583627c80144647e095afdf24ef8"} Sep 07 04:02:21.258723 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.258650 2555 scope.go:117] "RemoveContainer" containerID="74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0" Sep 07 04:02:21.258907 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.258803 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv" Sep 07 04:02:21.272785 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.272757 2555 scope.go:117] "RemoveContainer" containerID="74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0" Sep 07 04:02:21.273172 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:02:21.273147 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0\": container with ID starting with 74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0 not found: ID does not exist" containerID="74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0" Sep 07 04:02:21.273292 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.273179 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0"} err="failed to get container status \"74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0\": rpc error: code = NotFound desc = could not find container \"74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0\": container with ID starting with 74db2a84c85d8708c2063a1ac079c329e0ea9e48b8b59a53d9afc071137ee5e0 not found: ID does not exist" Sep 07 04:02:21.281868 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.281845 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv"] Sep 07 04:02:21.285150 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.285125 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d42cf5d-55984488cb-l44gv"] Sep 07 04:02:21.944545 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:21.944510 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a248f122-e72e-469a-9246-c1eb7fcee008" path="/var/lib/kubelet/pods/a248f122-e72e-469a-9246-c1eb7fcee008/volumes" Sep 07 04:02:22.668079 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.667712 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6"] Sep 07 04:02:22.668279 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668170 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" Sep 07 04:02:22.668279 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668185 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" Sep 07 04:02:22.668279 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668196 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ded062d-85fc-4aaa-b20d-54aea673370e" containerName="main" Sep 07 04:02:22.668279 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668201 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ded062d-85fc-4aaa-b20d-54aea673370e" containerName="main" Sep 07 04:02:22.668279 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668209 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" Sep 07 04:02:22.668279 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668215 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f342094c-43e2-401d-aa73-a2283812f587" containerName="istio-proxy" Sep 07 04:02:22.668279 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668243 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a248f122-e72e-469a-9246-c1eb7fcee008" containerName="maas-api" Sep 07 04:02:22.668279 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668250 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a248f122-e72e-469a-9246-c1eb7fcee008" containerName="maas-api" Sep 07 04:02:22.668694 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668308 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a248f122-e72e-469a-9246-c1eb7fcee008" containerName="maas-api" Sep 07 04:02:22.668694 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668322 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" Sep 07 04:02:22.668694 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668331 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ded062d-85fc-4aaa-b20d-54aea673370e" containerName="main" Sep 07 04:02:22.668694 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.668343 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" Sep 07 04:02:22.673097 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.673076 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.677500 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.677475 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-320efe75-openshift-default-dockercfg-bgw4t\"" Sep 07 04:02:22.688223 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.688200 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6"] Sep 07 04:02:22.725248 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.725212 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-envoy\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.725397 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.725262 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istiod-ca-cert\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.725397 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.725330 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-credential-socket\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.725397 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.725384 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-certs\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.725521 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.725471 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-socket\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.725521 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.725507 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-data\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.725604 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.725546 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-token\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.725604 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.725582 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhqh\" (UniqueName: \"kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-kube-api-access-7lhqh\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.725681 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.725619 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-podinfo\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827102 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827044 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-data\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827304 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827121 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-token\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827304 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827155 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhqh\" (UniqueName: \"kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-kube-api-access-7lhqh\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827304 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827180 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-podinfo\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827304 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827220 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-envoy\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827304 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827274 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istiod-ca-cert\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827569 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827338 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-credential-socket\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827569 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827371 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-certs\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827569 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827443 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-data\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827569 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827468 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-socket\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827863 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827838 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-socket\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.827947 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827869 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-certs\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.828012 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827991 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-credential-socket\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.828012 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.827996 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istiod-ca-cert\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.829409 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.829382 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-envoy\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.829812 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.829790 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-podinfo\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.838998 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.838580 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhqh\" (UniqueName: \"kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-kube-api-access-7lhqh\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.838998 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.838962 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-token\") pod \"e2e-derive-320efe75-openshift-default-7985f8745b-r47x6\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:22.986153 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:22.986079 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:23.287213 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:23.287092 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6"] Sep 07 04:02:23.289776 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:02:23.289751 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4b0fba_d271_44e2_9a1f_e8b8c27da434.slice/crio-ecd377c46fa0edb58c212873e3cc2635351688a4edeb3005c742e070085acdcf WatchSource:0}: Error finding container ecd377c46fa0edb58c212873e3cc2635351688a4edeb3005c742e070085acdcf: Status 404 returned error can't find the container with id ecd377c46fa0edb58c212873e3cc2635351688a4edeb3005c742e070085acdcf Sep 07 04:02:23.292127 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:23.292094 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:23.292204 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:23.292160 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:23.292204 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:23.292188 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:24.284774 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:24.284172 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" event={"ID":"ac4b0fba-d271-44e2-9a1f-e8b8c27da434","Type":"ContainerStarted","Data":"3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad"} Sep 07 04:02:24.285158 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:24.284846 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" event={"ID":"ac4b0fba-d271-44e2-9a1f-e8b8c27da434","Type":"ContainerStarted","Data":"ecd377c46fa0edb58c212873e3cc2635351688a4edeb3005c742e070085acdcf"} Sep 07 04:02:24.303869 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:24.303825 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" podStartSLOduration=2.30381146 podStartE2EDuration="2.30381146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:02:24.301679641 +0000 UTC m=+862.876545929" watchObservedRunningTime="2026-09-07 04:02:24.30381146 +0000 UTC m=+862.878677748" Sep 07 04:02:24.986599 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:24.986569 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:24.991640 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:24.991621 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:25.289727 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:25.289700 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:25.290676 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:25.290654 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:27.083114 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.083076 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw"] Sep 07 04:02:27.083647 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.083629 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" Sep 07 04:02:27.083647 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.083648 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97b3dc0-690b-4661-86b7-638f1e41cfb2" containerName="istio-proxy" Sep 07 04:02:27.087669 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.087646 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.089905 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.089885 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-320efe75-serving-cert\"" Sep 07 04:02:27.090223 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.089929 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-m9mvz\"" Sep 07 04:02:27.091020 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.091000 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 07 04:02:27.091142 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.091115 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 07 04:02:27.091354 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.091336 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 04:02:27.099261 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.099243 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw"] Sep 07 04:02:27.170048 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.170012 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-maas-api-tls\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.170244 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.170090 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w56w\" (UniqueName: \"kubernetes.io/projected/0198d581-b9b9-41cc-aa96-8758f15e7d7f-kube-api-access-9w56w\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.170244 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.170137 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-metrics-certs\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.170244 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.170154 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0198d581-b9b9-41cc-aa96-8758f15e7d7f-openshift-service-ca\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.271018 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.270987 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w56w\" (UniqueName: \"kubernetes.io/projected/0198d581-b9b9-41cc-aa96-8758f15e7d7f-kube-api-access-9w56w\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.271177 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.271030 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-metrics-certs\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.271177 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.271144 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0198d581-b9b9-41cc-aa96-8758f15e7d7f-openshift-service-ca\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.271373 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.271304 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-maas-api-tls\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.271790 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.271769 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0198d581-b9b9-41cc-aa96-8758f15e7d7f-openshift-service-ca\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.273483 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.273455 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-metrics-certs\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.273570 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.273508 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-maas-api-tls\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.278500 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.278475 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w56w\" (UniqueName: \"kubernetes.io/projected/0198d581-b9b9-41cc-aa96-8758f15e7d7f-kube-api-access-9w56w\") pod \"maas-api-e2e-derive-320efe75-7445f586bd-624qw\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.404256 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.404172 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:27.682975 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:27.680191 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw"] Sep 07 04:02:27.705359 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:02:27.683335 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0198d581_b9b9_41cc_aa96_8758f15e7d7f.slice/crio-ca0c2a4c4e047e2a0932f63a363e38eec313cc4945396a0fe20f6c89774b3fcd WatchSource:0}: Error finding container ca0c2a4c4e047e2a0932f63a363e38eec313cc4945396a0fe20f6c89774b3fcd: Status 404 returned error can't find the container with id ca0c2a4c4e047e2a0932f63a363e38eec313cc4945396a0fe20f6c89774b3fcd Sep 07 04:02:28.091263 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:28.091223 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:02:28.204802 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:28.204769 2555 patch_prober.go:28] interesting pod/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 07 04:02:28.204977 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:28.204836 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 07 04:02:28.304971 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:28.304942 2555 generic.go:356] "Generic (PLEG): container finished" podID="50d648ab-ba1e-4c70-98d4-543361d12798" containerID="49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf" exitCode=0 Sep 07 04:02:28.305123 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:28.305009 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" event={"ID":"50d648ab-ba1e-4c70-98d4-543361d12798","Type":"ContainerDied","Data":"49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf"} Sep 07 04:02:28.305509 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:28.305491 2555 scope.go:117] "RemoveContainer" containerID="49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf" Sep 07 04:02:28.306418 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:28.306391 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" event={"ID":"0198d581-b9b9-41cc-aa96-8758f15e7d7f","Type":"ContainerStarted","Data":"ca0c2a4c4e047e2a0932f63a363e38eec313cc4945396a0fe20f6c89774b3fcd"} Sep 07 04:02:28.306620 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:28.306594 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:28.306695 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:28.306677 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:28.306760 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:28.306727 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:29.318949 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:29.318903 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" event={"ID":"50d648ab-ba1e-4c70-98d4-543361d12798","Type":"ContainerStarted","Data":"c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd"} Sep 07 04:02:29.322981 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:29.322942 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" event={"ID":"0198d581-b9b9-41cc-aa96-8758f15e7d7f","Type":"ContainerStarted","Data":"9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b"} Sep 07 04:02:29.323613 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:29.323593 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:29.358840 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:29.358766 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" podStartSLOduration=1.401886202 podStartE2EDuration="2.358741848s" totalImagesPullingTime="956.855646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:02:29.356261122 +0000 UTC m=+867.931127411" watchObservedRunningTime="2026-09-07 04:02:29.358741848 +0000 UTC m=+867.933608139" Sep 07 04:02:29.743473 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:29.743432 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:29.743764 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:29.743479 2555 patch_prober.go:28] interesting pod/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 07 04:02:29.744305 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:29.744273 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 07 04:02:30.743432 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:30.743386 2555 patch_prober.go:28] interesting pod/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 07 04:02:30.743917 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:30.743471 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 07 04:02:31.742728 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:31.742692 2555 patch_prober.go:28] interesting pod/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 07 04:02:31.743012 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:31.742979 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 07 04:02:32.743505 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:32.743460 2555 patch_prober.go:28] interesting pod/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 07 04:02:32.743979 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:32.743561 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 07 04:02:33.743327 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:33.743281 2555 patch_prober.go:28] interesting pod/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 07 04:02:33.743521 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:33.743356 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 07 04:02:34.789982 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:34.789946 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:34.790466 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:34.790303 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:34.791259 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:34.791239 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:02:36.338997 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:36.338962 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:46.589949 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:46.589913 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:02:48.441677 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:48.441637 2555 generic.go:356] "Generic (PLEG): container finished" podID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerID="3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad" exitCode=0 Sep 07 04:02:48.442233 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:48.441710 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" event={"ID":"ac4b0fba-d271-44e2-9a1f-e8b8c27da434","Type":"ContainerDied","Data":"3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad"} Sep 07 04:02:48.442310 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:48.442254 2555 scope.go:117] "RemoveContainer" containerID="3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad" Sep 07 04:02:48.443406 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:48.443375 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:48.443484 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:48.443469 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:48.443526 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:48.443505 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:02:49.452606 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:49.452558 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" event={"ID":"ac4b0fba-d271-44e2-9a1f-e8b8c27da434","Type":"ContainerStarted","Data":"fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513"} Sep 07 04:02:49.986711 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:49.986674 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:49.986899 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:49.986876 2555 patch_prober.go:28] interesting pod/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 07 04:02:49.986945 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:49.986925 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 07 04:02:50.986933 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:50.986900 2555 patch_prober.go:28] interesting pod/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 07 04:02:50.987347 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:50.986954 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 07 04:02:51.986642 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:51.986600 2555 patch_prober.go:28] interesting pod/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 07 04:02:51.986853 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:51.986668 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 07 04:02:52.987257 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:52.987219 2555 patch_prober.go:28] interesting pod/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 07 04:02:52.987722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:52.987297 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 07 04:02:53.987232 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:53.987192 2555 patch_prober.go:28] interesting pod/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 07 04:02:53.987411 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:53.987260 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 07 04:02:54.991320 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:54.991289 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:54.991835 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:54.991656 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:54.992372 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:54.992346 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:02:57.479712 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.479680 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw"] Sep 07 04:02:57.480300 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.479926 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" podUID="0198d581-b9b9-41cc-aa96-8758f15e7d7f" containerName="maas-api" containerID="cri-o://9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b" gracePeriod=30 Sep 07 04:02:57.730181 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.730126 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:57.760335 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.760304 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w56w\" (UniqueName: \"kubernetes.io/projected/0198d581-b9b9-41cc-aa96-8758f15e7d7f-kube-api-access-9w56w\") pod \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " Sep 07 04:02:57.760511 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.760351 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-maas-api-tls\") pod \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " Sep 07 04:02:57.760511 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.760393 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-metrics-certs\") pod \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " Sep 07 04:02:57.760511 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.760480 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0198d581-b9b9-41cc-aa96-8758f15e7d7f-openshift-service-ca\") pod \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\" (UID: \"0198d581-b9b9-41cc-aa96-8758f15e7d7f\") " Sep 07 04:02:57.761026 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.760995 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0198d581-b9b9-41cc-aa96-8758f15e7d7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0198d581-b9b9-41cc-aa96-8758f15e7d7f" (UID: "0198d581-b9b9-41cc-aa96-8758f15e7d7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:02:57.762949 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.762922 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0198d581-b9b9-41cc-aa96-8758f15e7d7f-kube-api-access-9w56w" (OuterVolumeSpecName: "kube-api-access-9w56w") pod "0198d581-b9b9-41cc-aa96-8758f15e7d7f" (UID: "0198d581-b9b9-41cc-aa96-8758f15e7d7f"). InnerVolumeSpecName "kube-api-access-9w56w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:02:57.763446 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.763419 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0198d581-b9b9-41cc-aa96-8758f15e7d7f" (UID: "0198d581-b9b9-41cc-aa96-8758f15e7d7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:02:57.763631 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.763606 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0198d581-b9b9-41cc-aa96-8758f15e7d7f" (UID: "0198d581-b9b9-41cc-aa96-8758f15e7d7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:02:57.807818 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.807783 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6"] Sep 07 04:02:57.808105 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.808041 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" containerID="cri-o://fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513" gracePeriod=30 Sep 07 04:02:57.861525 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.861494 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0198d581-b9b9-41cc-aa96-8758f15e7d7f-openshift-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:57.861525 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.861521 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w56w\" (UniqueName: \"kubernetes.io/projected/0198d581-b9b9-41cc-aa96-8758f15e7d7f-kube-api-access-9w56w\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:57.861525 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.861531 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-maas-api-tls\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:57.861763 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:57.861539 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0198d581-b9b9-41cc-aa96-8758f15e7d7f-metrics-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:02:58.508295 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:58.508259 2555 generic.go:356] "Generic (PLEG): container finished" podID="0198d581-b9b9-41cc-aa96-8758f15e7d7f" containerID="9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b" exitCode=0 Sep 07 04:02:58.508749 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:58.508335 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" Sep 07 04:02:58.508749 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:58.508373 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" event={"ID":"0198d581-b9b9-41cc-aa96-8758f15e7d7f","Type":"ContainerDied","Data":"9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b"} Sep 07 04:02:58.508749 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:58.508422 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw" event={"ID":"0198d581-b9b9-41cc-aa96-8758f15e7d7f","Type":"ContainerDied","Data":"ca0c2a4c4e047e2a0932f63a363e38eec313cc4945396a0fe20f6c89774b3fcd"} Sep 07 04:02:58.508749 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:58.508454 2555 scope.go:117] "RemoveContainer" containerID="9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b" Sep 07 04:02:58.519774 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:58.519749 2555 scope.go:117] "RemoveContainer" containerID="9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b" Sep 07 04:02:58.520082 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:02:58.520036 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b\": container with ID starting with 9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b not found: ID does not exist" containerID="9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b" Sep 07 04:02:58.520161 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:58.520078 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b"} err="failed to get container status \"9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b\": rpc error: code = NotFound desc = could not find container \"9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b\": container with ID starting with 9a5b8dc226a1bcd9b72898aa00a4d9a7345b44aec90db1cab5e7372f91e9bb4b not found: ID does not exist" Sep 07 04:02:58.526295 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:58.526276 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw"] Sep 07 04:02:58.532934 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:58.532911 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-320efe75-7445f586bd-624qw"] Sep 07 04:02:59.937617 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:02:59.937581 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0198d581-b9b9-41cc-aa96-8758f15e7d7f" path="/var/lib/kubelet/pods/0198d581-b9b9-41cc-aa96-8758f15e7d7f/volumes" Sep 07 04:03:01.945555 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:01.945526 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:03:01.946087 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:01.946068 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:03:03.055165 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.055141 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:03:03.102865 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.102792 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-envoy\") pod \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " Sep 07 04:03:03.102865 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.102829 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-socket\") pod \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " Sep 07 04:03:03.103160 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.102878 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-podinfo\") pod \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " Sep 07 04:03:03.103160 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.102898 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istiod-ca-cert\") pod \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " Sep 07 04:03:03.103160 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.102956 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-credential-socket\") pod \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " Sep 07 04:03:03.103160 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.103000 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lhqh\" (UniqueName: \"kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-kube-api-access-7lhqh\") pod \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " Sep 07 04:03:03.103160 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.103032 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-token\") pod \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " Sep 07 04:03:03.103160 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.103083 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-certs\") pod \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " Sep 07 04:03:03.103160 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.103108 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-data\") pod \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\" (UID: \"ac4b0fba-d271-44e2-9a1f-e8b8c27da434\") " Sep 07 04:03:03.103536 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.103195 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac4b0fba-d271-44e2-9a1f-e8b8c27da434" (UID: "ac4b0fba-d271-44e2-9a1f-e8b8c27da434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:03.103536 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.103399 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac4b0fba-d271-44e2-9a1f-e8b8c27da434" (UID: "ac4b0fba-d271-44e2-9a1f-e8b8c27da434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:03.103536 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.103456 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:03.103704 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.103654 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac4b0fba-d271-44e2-9a1f-e8b8c27da434" (UID: "ac4b0fba-d271-44e2-9a1f-e8b8c27da434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:03.103704 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.103668 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac4b0fba-d271-44e2-9a1f-e8b8c27da434" (UID: "ac4b0fba-d271-44e2-9a1f-e8b8c27da434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:03.103854 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.103822 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac4b0fba-d271-44e2-9a1f-e8b8c27da434" (UID: "ac4b0fba-d271-44e2-9a1f-e8b8c27da434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:03:03.105232 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.105208 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac4b0fba-d271-44e2-9a1f-e8b8c27da434" (UID: "ac4b0fba-d271-44e2-9a1f-e8b8c27da434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:03.105419 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.105388 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-kube-api-access-7lhqh" (OuterVolumeSpecName: "kube-api-access-7lhqh") pod "ac4b0fba-d271-44e2-9a1f-e8b8c27da434" (UID: "ac4b0fba-d271-44e2-9a1f-e8b8c27da434"). InnerVolumeSpecName "kube-api-access-7lhqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:03:03.105564 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.105549 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac4b0fba-d271-44e2-9a1f-e8b8c27da434" (UID: "ac4b0fba-d271-44e2-9a1f-e8b8c27da434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:03:03.105647 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.105633 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac4b0fba-d271-44e2-9a1f-e8b8c27da434" (UID: "ac4b0fba-d271-44e2-9a1f-e8b8c27da434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:03:03.203995 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.203963 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:03.203995 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.203990 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:03.204218 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.204003 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:03.204218 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.204015 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lhqh\" (UniqueName: \"kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-kube-api-access-7lhqh\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:03.204218 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.204027 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:03.204218 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.204039 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:03.204218 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.204050 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:03.204218 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.204086 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4b0fba-d271-44e2-9a1f-e8b8c27da434-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:03.491818 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.491785 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:03:03.536211 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.536176 2555 generic.go:356] "Generic (PLEG): container finished" podID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerID="fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513" exitCode=0 Sep 07 04:03:03.536399 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.536220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" event={"ID":"ac4b0fba-d271-44e2-9a1f-e8b8c27da434","Type":"ContainerDied","Data":"fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513"} Sep 07 04:03:03.536399 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.536228 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" Sep 07 04:03:03.536399 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.536301 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6" event={"ID":"ac4b0fba-d271-44e2-9a1f-e8b8c27da434","Type":"ContainerDied","Data":"ecd377c46fa0edb58c212873e3cc2635351688a4edeb3005c742e070085acdcf"} Sep 07 04:03:03.536399 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.536333 2555 scope.go:117] "RemoveContainer" containerID="fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513" Sep 07 04:03:03.546917 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.546871 2555 scope.go:117] "RemoveContainer" containerID="3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad" Sep 07 04:03:03.555487 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.555468 2555 scope.go:117] "RemoveContainer" containerID="fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513" Sep 07 04:03:03.555742 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:03:03.555724 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513\": container with ID starting with fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513 not found: ID does not exist" containerID="fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513" Sep 07 04:03:03.555799 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.555750 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513"} err="failed to get container status \"fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513\": rpc error: code = NotFound desc = could not find container \"fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513\": container with ID starting with fb86905e44b736e89d7e0ec5cb40741f76684d29ac7969d6587e64e23b5db513 not found: ID does not exist" Sep 07 04:03:03.555799 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.555767 2555 scope.go:117] "RemoveContainer" containerID="3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad" Sep 07 04:03:03.556032 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:03:03.556016 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad\": container with ID starting with 3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad not found: ID does not exist" containerID="3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad" Sep 07 04:03:03.556091 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.556039 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad"} err="failed to get container status \"3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad\": rpc error: code = NotFound desc = could not find container \"3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad\": container with ID starting with 3360debfad34dfc735dd9ff46ac6344665361979e86c01fb878b305d5ef500ad not found: ID does not exist" Sep 07 04:03:03.559166 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.559144 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6"] Sep 07 04:03:03.567078 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.567040 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-320efe75-openshift-default-7985f8745b-r47x6"] Sep 07 04:03:03.940018 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:03.939927 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" path="/var/lib/kubelet/pods/ac4b0fba-d271-44e2-9a1f-e8b8c27da434/volumes" Sep 07 04:03:11.717888 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:11.717851 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh"] Sep 07 04:03:11.718432 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:11.718208 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" containerID="cri-o://c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd" gracePeriod=30 Sep 07 04:03:12.461199 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.461163 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp"] Sep 07 04:03:12.461613 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.461591 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" Sep 07 04:03:12.461613 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.461612 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" Sep 07 04:03:12.461799 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.461632 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0198d581-b9b9-41cc-aa96-8758f15e7d7f" containerName="maas-api" Sep 07 04:03:12.461799 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.461642 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0198d581-b9b9-41cc-aa96-8758f15e7d7f" containerName="maas-api" Sep 07 04:03:12.461799 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.461654 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" Sep 07 04:03:12.461799 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.461663 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" Sep 07 04:03:12.461799 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.461796 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0198d581-b9b9-41cc-aa96-8758f15e7d7f" containerName="maas-api" Sep 07 04:03:12.462047 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.461806 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" Sep 07 04:03:12.462047 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.461812 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac4b0fba-d271-44e2-9a1f-e8b8c27da434" containerName="istio-proxy" Sep 07 04:03:12.466599 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.466569 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.469139 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.469113 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-860c6d-openshift-default-dockercfg-s58wk\"" Sep 07 04:03:12.478336 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.478316 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp"] Sep 07 04:03:12.491720 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.491691 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-data\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.491884 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.491726 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wctdz\" (UniqueName: \"kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-kube-api-access-wctdz\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.491884 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.491745 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istiod-ca-cert\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.491884 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.491773 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-token\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.491884 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.491845 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-certs\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.491884 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.491863 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-podinfo\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.492118 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.491892 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-credential-socket\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.492118 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.491940 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-socket\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.492118 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.492024 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-envoy\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593047 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593009 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-data\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593047 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593069 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wctdz\" (UniqueName: \"kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-kube-api-access-wctdz\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593330 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593101 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istiod-ca-cert\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593330 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593150 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-token\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593330 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593187 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-certs\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593330 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593216 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-podinfo\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593330 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593262 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-credential-socket\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593330 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593292 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-socket\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593650 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593340 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-envoy\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593650 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593468 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-data\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593763 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593663 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-certs\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593763 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593674 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-credential-socket\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593763 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593716 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-socket\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.593913 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.593856 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istiod-ca-cert\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.595804 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.595779 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-envoy\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.595998 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.595982 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-podinfo\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.600641 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.600613 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-token\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.600847 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.600823 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wctdz\" (UniqueName: \"kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-kube-api-access-wctdz\") pod \"e2e-noten-860c6d-openshift-default-57cd778746-j8wcp\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:12.778335 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:12.778238 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:13.054810 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:13.054780 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp"] Sep 07 04:03:13.057959 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:03:13.057921 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32612fd_dd1b_4cf7_8b17_f43a35934c60.slice/crio-44c0ebaf95834af6492803e1cf8ebb0891dfbef7476753c2bef3eb9512e13774 WatchSource:0}: Error finding container 44c0ebaf95834af6492803e1cf8ebb0891dfbef7476753c2bef3eb9512e13774: Status 404 returned error can't find the container with id 44c0ebaf95834af6492803e1cf8ebb0891dfbef7476753c2bef3eb9512e13774 Sep 07 04:03:13.060470 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:13.060433 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:03:13.060561 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:13.060541 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:03:13.060626 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:13.060585 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:03:13.592791 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:13.592755 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" event={"ID":"e32612fd-dd1b-4cf7-8b17-f43a35934c60","Type":"ContainerStarted","Data":"65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8"} Sep 07 04:03:13.592982 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:13.592805 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" event={"ID":"e32612fd-dd1b-4cf7-8b17-f43a35934c60","Type":"ContainerStarted","Data":"44c0ebaf95834af6492803e1cf8ebb0891dfbef7476753c2bef3eb9512e13774"} Sep 07 04:03:13.614414 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:13.614347 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" podStartSLOduration=1.614332465 podStartE2EDuration="1.614332465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:03:13.61175364 +0000 UTC m=+912.186619932" watchObservedRunningTime="2026-09-07 04:03:13.614332465 +0000 UTC m=+912.189198753" Sep 07 04:03:13.779249 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:13.779205 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:14.783903 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:14.783873 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:15.602981 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:15.602953 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:15.603944 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:15.603908 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:16.971318 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:16.971293 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:03:17.038078 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.037972 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-credential-socket\") pod \"50d648ab-ba1e-4c70-98d4-543361d12798\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " Sep 07 04:03:17.038078 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038016 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-data\") pod \"50d648ab-ba1e-4c70-98d4-543361d12798\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " Sep 07 04:03:17.038078 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038040 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50d648ab-ba1e-4c70-98d4-543361d12798-istio-podinfo\") pod \"50d648ab-ba1e-4c70-98d4-543361d12798\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " Sep 07 04:03:17.038078 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038072 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-socket\") pod \"50d648ab-ba1e-4c70-98d4-543361d12798\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " Sep 07 04:03:17.038415 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038090 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50d648ab-ba1e-4c70-98d4-543361d12798-istiod-ca-cert\") pod \"50d648ab-ba1e-4c70-98d4-543361d12798\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " Sep 07 04:03:17.038415 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038167 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-certs\") pod \"50d648ab-ba1e-4c70-98d4-543361d12798\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " Sep 07 04:03:17.038415 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038202 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzkdc\" (UniqueName: \"kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-kube-api-access-xzkdc\") pod \"50d648ab-ba1e-4c70-98d4-543361d12798\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " Sep 07 04:03:17.038415 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038310 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-data" (OuterVolumeSpecName: "istio-data") pod "50d648ab-ba1e-4c70-98d4-543361d12798" (UID: "50d648ab-ba1e-4c70-98d4-543361d12798"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:17.038415 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038336 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-istio-token\") pod \"50d648ab-ba1e-4c70-98d4-543361d12798\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " Sep 07 04:03:17.038415 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038370 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-envoy\") pod \"50d648ab-ba1e-4c70-98d4-543361d12798\" (UID: \"50d648ab-ba1e-4c70-98d4-543361d12798\") " Sep 07 04:03:17.038415 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038314 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "50d648ab-ba1e-4c70-98d4-543361d12798" (UID: "50d648ab-ba1e-4c70-98d4-543361d12798"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:17.038756 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038399 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "50d648ab-ba1e-4c70-98d4-543361d12798" (UID: "50d648ab-ba1e-4c70-98d4-543361d12798"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:17.038756 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038584 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d648ab-ba1e-4c70-98d4-543361d12798-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "50d648ab-ba1e-4c70-98d4-543361d12798" (UID: "50d648ab-ba1e-4c70-98d4-543361d12798"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:03:17.038756 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038726 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "50d648ab-ba1e-4c70-98d4-543361d12798" (UID: "50d648ab-ba1e-4c70-98d4-543361d12798"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:17.038914 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038786 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:17.038914 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038806 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:17.038914 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038820 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:17.038914 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.038833 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50d648ab-ba1e-4c70-98d4-543361d12798-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:17.040453 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.040428 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-kube-api-access-xzkdc" (OuterVolumeSpecName: "kube-api-access-xzkdc") pod "50d648ab-ba1e-4c70-98d4-543361d12798" (UID: "50d648ab-ba1e-4c70-98d4-543361d12798"). InnerVolumeSpecName "kube-api-access-xzkdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:03:17.040453 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.040444 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-istio-token" (OuterVolumeSpecName: "istio-token") pod "50d648ab-ba1e-4c70-98d4-543361d12798" (UID: "50d648ab-ba1e-4c70-98d4-543361d12798"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:03:17.040763 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.040742 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50d648ab-ba1e-4c70-98d4-543361d12798-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "50d648ab-ba1e-4c70-98d4-543361d12798" (UID: "50d648ab-ba1e-4c70-98d4-543361d12798"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:03:17.040806 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.040780 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "50d648ab-ba1e-4c70-98d4-543361d12798" (UID: "50d648ab-ba1e-4c70-98d4-543361d12798"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:17.140153 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.140117 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50d648ab-ba1e-4c70-98d4-543361d12798-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:17.140153 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.140144 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:17.140153 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.140153 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzkdc\" (UniqueName: \"kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-kube-api-access-xzkdc\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:17.140153 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.140163 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50d648ab-ba1e-4c70-98d4-543361d12798-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:17.140415 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.140171 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50d648ab-ba1e-4c70-98d4-543361d12798-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:17.615337 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.615298 2555 generic.go:356] "Generic (PLEG): container finished" podID="50d648ab-ba1e-4c70-98d4-543361d12798" containerID="c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd" exitCode=0 Sep 07 04:03:17.615525 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.615353 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" Sep 07 04:03:17.615525 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.615382 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" event={"ID":"50d648ab-ba1e-4c70-98d4-543361d12798","Type":"ContainerDied","Data":"c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd"} Sep 07 04:03:17.615525 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.615439 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh" event={"ID":"50d648ab-ba1e-4c70-98d4-543361d12798","Type":"ContainerDied","Data":"cd84f20bc38bc8b0f086cff83c171407b90b4c90904f529bbb41b7d719bd09a1"} Sep 07 04:03:17.615525 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.615464 2555 scope.go:117] "RemoveContainer" containerID="c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd" Sep 07 04:03:17.625017 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.625001 2555 scope.go:117] "RemoveContainer" containerID="49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf" Sep 07 04:03:17.635271 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.635247 2555 scope.go:117] "RemoveContainer" containerID="c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd" Sep 07 04:03:17.635602 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:03:17.635582 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd\": container with ID starting with c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd not found: ID does not exist" containerID="c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd" Sep 07 04:03:17.635682 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.635616 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd"} err="failed to get container status \"c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd\": rpc error: code = NotFound desc = could not find container \"c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd\": container with ID starting with c17573235e246e11503a798197642eebaf4a8c47b5d523631f37490777a17dcd not found: ID does not exist" Sep 07 04:03:17.635682 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.635641 2555 scope.go:117] "RemoveContainer" containerID="49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf" Sep 07 04:03:17.635908 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:03:17.635891 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf\": container with ID starting with 49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf not found: ID does not exist" containerID="49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf" Sep 07 04:03:17.635964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.635915 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf"} err="failed to get container status \"49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf\": rpc error: code = NotFound desc = could not find container \"49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf\": container with ID starting with 49670ffcbdf0df1a10c2bd6995c743d753dd91d42c6a7212a5957a0f9c6392bf not found: ID does not exist" Sep 07 04:03:17.640378 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.640357 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh"] Sep 07 04:03:17.645669 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.645649 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-a84f90-openshift-default-74f8c4556d-bzcrh"] Sep 07 04:03:17.937415 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:17.937329 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" path="/var/lib/kubelet/pods/50d648ab-ba1e-4c70-98d4-543361d12798/volumes" Sep 07 04:03:20.985251 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:20.985212 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:03:33.186343 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:33.186303 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:03:44.269290 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:44.269256 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp"] Sep 07 04:03:44.269762 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:44.269499 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" podUID="e32612fd-dd1b-4cf7-8b17-f43a35934c60" containerName="istio-proxy" containerID="cri-o://65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8" gracePeriod=30 Sep 07 04:03:45.605349 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:45.605311 2555 patch_prober.go:28] interesting pod/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 04:03:45.605736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:45.605377 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" podUID="e32612fd-dd1b-4cf7-8b17-f43a35934c60" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 04:03:49.515544 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.515522 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:49.531351 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531298 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-certs\") pod \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " Sep 07 04:03:49.531430 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531360 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-envoy\") pod \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " Sep 07 04:03:49.531430 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531380 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-credential-socket\") pod \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " Sep 07 04:03:49.531430 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531400 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-socket\") pod \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " Sep 07 04:03:49.531430 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531422 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wctdz\" (UniqueName: \"kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-kube-api-access-wctdz\") pod \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " Sep 07 04:03:49.531610 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531478 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istiod-ca-cert\") pod \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " Sep 07 04:03:49.531610 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531505 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-podinfo\") pod \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " Sep 07 04:03:49.531610 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531538 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-data\") pod \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " Sep 07 04:03:49.531610 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531553 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e32612fd-dd1b-4cf7-8b17-f43a35934c60" (UID: "e32612fd-dd1b-4cf7-8b17-f43a35934c60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:49.531610 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531561 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-token\") pod \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\" (UID: \"e32612fd-dd1b-4cf7-8b17-f43a35934c60\") " Sep 07 04:03:49.531865 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531677 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e32612fd-dd1b-4cf7-8b17-f43a35934c60" (UID: "e32612fd-dd1b-4cf7-8b17-f43a35934c60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:49.531865 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531690 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e32612fd-dd1b-4cf7-8b17-f43a35934c60" (UID: "e32612fd-dd1b-4cf7-8b17-f43a35934c60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:49.531992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531861 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-data" (OuterVolumeSpecName: "istio-data") pod "e32612fd-dd1b-4cf7-8b17-f43a35934c60" (UID: "e32612fd-dd1b-4cf7-8b17-f43a35934c60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:49.531992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531893 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e32612fd-dd1b-4cf7-8b17-f43a35934c60" (UID: "e32612fd-dd1b-4cf7-8b17-f43a35934c60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:03:49.531992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531921 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:49.531992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531941 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:49.531992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531957 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:49.531992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.531972 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:49.534557 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.534013 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-token" (OuterVolumeSpecName: "istio-token") pod "e32612fd-dd1b-4cf7-8b17-f43a35934c60" (UID: "e32612fd-dd1b-4cf7-8b17-f43a35934c60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:03:49.534557 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.534107 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-kube-api-access-wctdz" (OuterVolumeSpecName: "kube-api-access-wctdz") pod "e32612fd-dd1b-4cf7-8b17-f43a35934c60" (UID: "e32612fd-dd1b-4cf7-8b17-f43a35934c60"). InnerVolumeSpecName "kube-api-access-wctdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:03:49.534557 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.534447 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e32612fd-dd1b-4cf7-8b17-f43a35934c60" (UID: "e32612fd-dd1b-4cf7-8b17-f43a35934c60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:03:49.535503 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.535467 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e32612fd-dd1b-4cf7-8b17-f43a35934c60" (UID: "e32612fd-dd1b-4cf7-8b17-f43a35934c60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:03:49.633022 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.632975 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:49.633022 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.633019 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wctdz\" (UniqueName: \"kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-kube-api-access-wctdz\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:49.633272 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.633034 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:49.633272 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.633048 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:49.633272 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.633092 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32612fd-dd1b-4cf7-8b17-f43a35934c60-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:03:49.786493 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.786385 2555 generic.go:356] "Generic (PLEG): container finished" podID="e32612fd-dd1b-4cf7-8b17-f43a35934c60" containerID="65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8" exitCode=0 Sep 07 04:03:49.786493 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.786425 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" Sep 07 04:03:49.786493 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.786481 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" event={"ID":"e32612fd-dd1b-4cf7-8b17-f43a35934c60","Type":"ContainerDied","Data":"65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8"} Sep 07 04:03:49.786793 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.786533 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp" event={"ID":"e32612fd-dd1b-4cf7-8b17-f43a35934c60","Type":"ContainerDied","Data":"44c0ebaf95834af6492803e1cf8ebb0891dfbef7476753c2bef3eb9512e13774"} Sep 07 04:03:49.786793 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.786556 2555 scope.go:117] "RemoveContainer" containerID="65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8" Sep 07 04:03:49.797558 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.797541 2555 scope.go:117] "RemoveContainer" containerID="65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8" Sep 07 04:03:49.797810 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:03:49.797793 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8\": container with ID starting with 65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8 not found: ID does not exist" containerID="65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8" Sep 07 04:03:49.797878 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.797818 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8"} err="failed to get container status \"65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8\": rpc error: code = NotFound desc = could not find container \"65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8\": container with ID starting with 65d5fc04d02f7bc7d6ea4a03e1cb1959ae049640e19f5fd74a4c1ca4181e3fc8 not found: ID does not exist" Sep 07 04:03:49.809571 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.809550 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp"] Sep 07 04:03:49.813593 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.813573 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-860c6d-openshift-default-57cd778746-j8wcp"] Sep 07 04:03:49.939509 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:03:49.939475 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e32612fd-dd1b-4cf7-8b17-f43a35934c60" path="/var/lib/kubelet/pods/e32612fd-dd1b-4cf7-8b17-f43a35934c60/volumes" Sep 07 04:04:04.298175 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.298141 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp"] Sep 07 04:04:04.298644 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.298504 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" Sep 07 04:04:04.298644 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.298516 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" Sep 07 04:04:04.298644 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.298530 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e32612fd-dd1b-4cf7-8b17-f43a35934c60" containerName="istio-proxy" Sep 07 04:04:04.298644 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.298536 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e32612fd-dd1b-4cf7-8b17-f43a35934c60" containerName="istio-proxy" Sep 07 04:04:04.298644 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.298543 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" Sep 07 04:04:04.298644 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.298549 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" Sep 07 04:04:04.298644 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.298611 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" Sep 07 04:04:04.298644 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.298622 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e32612fd-dd1b-4cf7-8b17-f43a35934c60" containerName="istio-proxy" Sep 07 04:04:04.302692 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.302675 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp" Sep 07 04:04:04.306985 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.306959 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 04:04:04.307169 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.306959 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs4ff\"" Sep 07 04:04:04.309775 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.309750 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp"] Sep 07 04:04:04.370254 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.370222 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33ae947-714c-42aa-8997-8a2afdb479aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp\" (UID: \"b33ae947-714c-42aa-8997-8a2afdb479aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp" Sep 07 04:04:04.470735 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.470699 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33ae947-714c-42aa-8997-8a2afdb479aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp\" (UID: \"b33ae947-714c-42aa-8997-8a2afdb479aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp" Sep 07 04:04:04.471397 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.471370 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33ae947-714c-42aa-8997-8a2afdb479aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp\" (UID: \"b33ae947-714c-42aa-8997-8a2afdb479aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp" Sep 07 04:04:04.614917 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.614833 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp" Sep 07 04:04:04.884175 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:04.884152 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp"] Sep 07 04:04:04.886282 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:04:04.886251 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33ae947_714c_42aa_8997_8a2afdb479aa.slice/crio-b6af38a1443a66ae76cd1c1044157e261a51df0f90c7c40049caa01c85547d43 WatchSource:0}: Error finding container b6af38a1443a66ae76cd1c1044157e261a51df0f90c7c40049caa01c85547d43: Status 404 returned error can't find the container with id b6af38a1443a66ae76cd1c1044157e261a51df0f90c7c40049caa01c85547d43 Sep 07 04:04:05.869889 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:05.869851 2555 generic.go:356] "Generic (PLEG): container finished" podID="b33ae947-714c-42aa-8997-8a2afdb479aa" containerID="d3561225f63cc836809abfd472d04f6b2f2f168439a25a720e8d26587493e0eb" exitCode=0 Sep 07 04:04:05.870296 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:05.869913 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp" event={"ID":"b33ae947-714c-42aa-8997-8a2afdb479aa","Type":"ContainerDied","Data":"d3561225f63cc836809abfd472d04f6b2f2f168439a25a720e8d26587493e0eb"} Sep 07 04:04:05.870296 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:05.869952 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp" event={"ID":"b33ae947-714c-42aa-8997-8a2afdb479aa","Type":"ContainerStarted","Data":"b6af38a1443a66ae76cd1c1044157e261a51df0f90c7c40049caa01c85547d43"} Sep 07 04:04:07.005629 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:07.005608 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp" Sep 07 04:04:07.095525 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:07.095492 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33ae947-714c-42aa-8997-8a2afdb479aa-maas-api-service-ca\") pod \"b33ae947-714c-42aa-8997-8a2afdb479aa\" (UID: \"b33ae947-714c-42aa-8997-8a2afdb479aa\") " Sep 07 04:04:07.095863 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:07.095838 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33ae947-714c-42aa-8997-8a2afdb479aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b33ae947-714c-42aa-8997-8a2afdb479aa" (UID: "b33ae947-714c-42aa-8997-8a2afdb479aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:04:07.197088 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:07.196978 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33ae947-714c-42aa-8997-8a2afdb479aa-maas-api-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:04:07.880868 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:07.880840 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp" Sep 07 04:04:07.880868 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:07.880857 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp" event={"ID":"b33ae947-714c-42aa-8997-8a2afdb479aa","Type":"ContainerDied","Data":"b6af38a1443a66ae76cd1c1044157e261a51df0f90c7c40049caa01c85547d43"} Sep 07 04:04:07.881104 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:07.880901 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6af38a1443a66ae76cd1c1044157e261a51df0f90c7c40049caa01c85547d43" Sep 07 04:04:07.983603 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:07.983566 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:04:14.838772 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:14.838738 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp"] Sep 07 04:04:14.841909 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:14.841883 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1038cd-f4tnp"] Sep 07 04:04:15.936528 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:15.936489 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b33ae947-714c-42aa-8997-8a2afdb479aa" path="/var/lib/kubelet/pods/b33ae947-714c-42aa-8997-8a2afdb479aa/volumes" Sep 07 04:04:19.907107 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:19.907071 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:04:29.188139 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:29.188041 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:04:37.995285 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:37.995246 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:04:40.352240 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.352173 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7"] Sep 07 04:04:40.352694 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.352683 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33ae947-714c-42aa-8997-8a2afdb479aa" containerName="revoke-keys" Sep 07 04:04:40.352754 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.352700 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33ae947-714c-42aa-8997-8a2afdb479aa" containerName="revoke-keys" Sep 07 04:04:40.352796 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.352779 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33ae947-714c-42aa-8997-8a2afdb479aa" containerName="revoke-keys" Sep 07 04:04:40.352796 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.352788 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d648ab-ba1e-4c70-98d4-543361d12798" containerName="istio-proxy" Sep 07 04:04:40.356137 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.356117 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.358567 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.358507 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b8cea6-openshift-default-dockercfg-v2bgz\"" Sep 07 04:04:40.366412 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.366389 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7"] Sep 07 04:04:40.381642 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.381617 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hckqh\" (UniqueName: \"kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-kube-api-access-hckqh\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.381794 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.381654 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-socket\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.381794 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.381714 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-data\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.381794 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.381756 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-token\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.381968 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.381795 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-credential-socket\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.381968 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.381811 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ee7e09-c93d-49fb-a581-4523b390964a-istiod-ca-cert\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.381968 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.381848 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-envoy\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.381968 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.381865 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-podinfo\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.382207 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.381956 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-certs\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483115 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483087 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-envoy\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483297 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483122 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-podinfo\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483297 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483171 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-certs\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483297 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483211 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hckqh\" (UniqueName: \"kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-kube-api-access-hckqh\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483297 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483250 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-socket\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483297 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483289 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-data\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483557 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483317 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-token\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483557 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483355 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-credential-socket\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483557 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483378 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ee7e09-c93d-49fb-a581-4523b390964a-istiod-ca-cert\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483727 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483683 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-certs\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483786 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483688 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-socket\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.483948 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.483926 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-data\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.484070 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.484039 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-credential-socket\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.484138 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.484116 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ee7e09-c93d-49fb-a581-4523b390964a-istiod-ca-cert\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.485544 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.485518 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-envoy\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.485861 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.485841 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-podinfo\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.492217 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.492194 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-token\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.492494 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.492477 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hckqh\" (UniqueName: \"kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-kube-api-access-hckqh\") pod \"e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.669265 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.669228 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:40.941141 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.941115 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7"] Sep 07 04:04:40.943244 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:04:40.943220 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ee7e09_c93d_49fb_a581_4523b390964a.slice/crio-569c55529fab0b6cb123b3a15eea0ee141b96b50e694a0f4519fa184ef99e5c0 WatchSource:0}: Error finding container 569c55529fab0b6cb123b3a15eea0ee141b96b50e694a0f4519fa184ef99e5c0: Status 404 returned error can't find the container with id 569c55529fab0b6cb123b3a15eea0ee141b96b50e694a0f4519fa184ef99e5c0 Sep 07 04:04:40.945303 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.945272 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:04:40.945376 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.945336 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:04:40.945376 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:40.945363 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:04:41.050629 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:41.050593 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" event={"ID":"b6ee7e09-c93d-49fb-a581-4523b390964a","Type":"ContainerStarted","Data":"e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227"} Sep 07 04:04:41.050833 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:41.050660 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" event={"ID":"b6ee7e09-c93d-49fb-a581-4523b390964a","Type":"ContainerStarted","Data":"569c55529fab0b6cb123b3a15eea0ee141b96b50e694a0f4519fa184ef99e5c0"} Sep 07 04:04:41.068652 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:41.068599 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" podStartSLOduration=1.068584255 podStartE2EDuration="1.068584255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:04:41.067296729 +0000 UTC m=+999.642163021" watchObservedRunningTime="2026-09-07 04:04:41.068584255 +0000 UTC m=+999.643450608" Sep 07 04:04:41.670271 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:41.670233 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:42.675594 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:42.675556 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:43.062101 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:43.062071 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:43.062895 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:43.062875 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:47.107077 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:47.102401 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:04:58.064164 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:58.064127 2555 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 04:04:58.064565 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:58.064197 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 04:04:58.148989 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:58.148955 2555 generic.go:356] "Generic (PLEG): container finished" podID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerID="e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227" exitCode=0 Sep 07 04:04:58.149174 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:58.149027 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" event={"ID":"b6ee7e09-c93d-49fb-a581-4523b390964a","Type":"ContainerDied","Data":"e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227"} Sep 07 04:04:58.149456 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:58.149438 2555 scope.go:117] "RemoveContainer" containerID="e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227" Sep 07 04:04:58.150471 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:58.150446 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:04:58.150531 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:58.150522 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:04:58.150572 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:58.150556 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:04:59.159691 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:59.159651 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" event={"ID":"b6ee7e09-c93d-49fb-a581-4523b390964a","Type":"ContainerStarted","Data":"d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1"} Sep 07 04:04:59.669998 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:59.669955 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:04:59.670334 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:59.670308 2555 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 04:04:59.670451 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:04:59.670364 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 04:05:00.669778 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:00.669738 2555 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 04:05:00.670198 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:00.669792 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 04:05:01.670453 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:01.670409 2555 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 04:05:01.670850 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:01.670471 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 04:05:02.669992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:02.669953 2555 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 04:05:02.670214 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:02.670033 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 04:05:03.670677 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:03.670630 2555 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 04:05:03.671103 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:03.670698 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 04:05:04.673712 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:04.673679 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:05:04.674142 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:04.673942 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:05:04.675023 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:04.675006 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:05:12.827974 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.827936 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9"] Sep 07 04:05:12.832072 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.832039 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:12.834463 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.834405 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 04:05:12.834829 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.834804 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-434d57-serving-cert\"" Sep 07 04:05:12.835567 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.835542 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-m9mvz\"" Sep 07 04:05:12.835679 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.835662 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 07 04:05:12.835748 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.835666 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 07 04:05:12.841096 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.840942 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9"] Sep 07 04:05:12.985294 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.985256 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0fa4d5-1a1b-4523-93a5-66208522223d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:12.985464 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.985372 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:12.985464 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.985408 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:12.985464 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:12.985433 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvwn\" (UniqueName: \"kubernetes.io/projected/3f0fa4d5-1a1b-4523-93a5-66208522223d-kube-api-access-7qvwn\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:13.086334 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.086252 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:13.086334 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.086285 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:13.086334 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.086308 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvwn\" (UniqueName: \"kubernetes.io/projected/3f0fa4d5-1a1b-4523-93a5-66208522223d-kube-api-access-7qvwn\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:13.086639 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.086353 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0fa4d5-1a1b-4523-93a5-66208522223d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:13.086946 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.086921 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0fa4d5-1a1b-4523-93a5-66208522223d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:13.088713 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.088693 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:13.088815 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.088725 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:13.094311 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.094286 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvwn\" (UniqueName: \"kubernetes.io/projected/3f0fa4d5-1a1b-4523-93a5-66208522223d-kube-api-access-7qvwn\") pod \"maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:13.145562 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.145529 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:13.423975 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.423952 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9"] Sep 07 04:05:13.426332 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:05:13.426300 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0fa4d5_1a1b_4523_93a5_66208522223d.slice/crio-6cbc61be6a2c35fe52fa482f32f8fededdad73ff047b535c2d5f5bbed693a25f WatchSource:0}: Error finding container 6cbc61be6a2c35fe52fa482f32f8fededdad73ff047b535c2d5f5bbed693a25f: Status 404 returned error can't find the container with id 6cbc61be6a2c35fe52fa482f32f8fededdad73ff047b535c2d5f5bbed693a25f Sep 07 04:05:13.427568 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:13.427548 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 04:05:14.239275 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:14.239237 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" event={"ID":"3f0fa4d5-1a1b-4523-93a5-66208522223d","Type":"ContainerStarted","Data":"d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603"} Sep 07 04:05:14.239751 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:14.239293 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" event={"ID":"3f0fa4d5-1a1b-4523-93a5-66208522223d","Type":"ContainerStarted","Data":"6cbc61be6a2c35fe52fa482f32f8fededdad73ff047b535c2d5f5bbed693a25f"} Sep 07 04:05:14.239751 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:14.239325 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:14.256734 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:14.256625 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" podStartSLOduration=1.684591423 podStartE2EDuration="2.256608366s" totalImagesPullingTime="572.016943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:05:14.254366666 +0000 UTC m=+1032.829232956" watchObservedRunningTime="2026-09-07 04:05:14.256608366 +0000 UTC m=+1032.831474654" Sep 07 04:05:17.696458 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:17.696414 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:05:20.250046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:20.250015 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:05:28.685582 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:28.685547 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:05:48.187879 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:05:48.187838 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:06:11.329917 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.329837 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp"] Sep 07 04:06:11.333972 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.333937 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.336698 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.336671 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a89024-serving-cert\"" Sep 07 04:06:11.353959 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.353927 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp"] Sep 07 04:06:11.435476 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.435439 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.435706 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.435491 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.435777 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.435716 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd02547-b07d-4d83-ba95-6a586bc74dde-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.435841 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.435777 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwct\" (UniqueName: \"kubernetes.io/projected/3cd02547-b07d-4d83-ba95-6a586bc74dde-kube-api-access-hxwct\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.540113 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.540080 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd02547-b07d-4d83-ba95-6a586bc74dde-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.540314 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.540134 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwct\" (UniqueName: \"kubernetes.io/projected/3cd02547-b07d-4d83-ba95-6a586bc74dde-kube-api-access-hxwct\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.540314 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.540169 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.540314 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.540199 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.540810 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:06:11.540783 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-serving-cert: secret "maas-api-e2e-shared-b-w2-a89024-serving-cert" not found Sep 07 04:06:11.540946 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:06:11.540932 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-maas-api-tls podName:3cd02547-b07d-4d83-ba95-6a586bc74dde nodeName:}" failed. No retries permitted until 2026-09-07 04:06:12.040900122 +0000 UTC m=+1090.615766400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" (UID: "3cd02547-b07d-4d83-ba95-6a586bc74dde") : secret "maas-api-e2e-shared-b-w2-a89024-serving-cert" not found Sep 07 04:06:11.543578 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.543547 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.543698 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.543607 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd02547-b07d-4d83-ba95-6a586bc74dde-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:11.552193 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:11.552167 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwct\" (UniqueName: \"kubernetes.io/projected/3cd02547-b07d-4d83-ba95-6a586bc74dde-kube-api-access-hxwct\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:12.045546 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:12.045498 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:12.048295 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:12.048259 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:12.245746 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:12.245713 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:12.517929 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:12.517865 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp"] Sep 07 04:06:12.520283 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:06:12.520258 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd02547_b07d_4d83_ba95_6a586bc74dde.slice/crio-19bf1fe202ca4506d407783c701e19656dce9f8298e57c75617c3a177f4d9b5c WatchSource:0}: Error finding container 19bf1fe202ca4506d407783c701e19656dce9f8298e57c75617c3a177f4d9b5c: Status 404 returned error can't find the container with id 19bf1fe202ca4506d407783c701e19656dce9f8298e57c75617c3a177f4d9b5c Sep 07 04:06:12.550446 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:12.550365 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" event={"ID":"3cd02547-b07d-4d83-ba95-6a586bc74dde","Type":"ContainerStarted","Data":"19bf1fe202ca4506d407783c701e19656dce9f8298e57c75617c3a177f4d9b5c"} Sep 07 04:06:13.556695 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:13.556656 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" event={"ID":"3cd02547-b07d-4d83-ba95-6a586bc74dde","Type":"ContainerStarted","Data":"0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb"} Sep 07 04:06:13.557108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:13.556737 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:13.573643 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:13.573598 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" podStartSLOduration=2.138221966 podStartE2EDuration="2.573580695s" totalImagesPullingTime="435.358729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:06:13.571738944 +0000 UTC m=+1092.146605250" watchObservedRunningTime="2026-09-07 04:06:13.573580695 +0000 UTC m=+1092.148446983" Sep 07 04:06:16.793097 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:16.793048 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:06:19.566861 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:19.566832 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:06:27.694041 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:27.694006 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:06:35.759157 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:35.759124 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7353-140202966496576"] Sep 07 04:06:35.762642 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:35.762626 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:35.764834 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:35.764812 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhk6n\"" Sep 07 04:06:35.768164 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:35.768141 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-140202966496576"] Sep 07 04:06:35.864194 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:35.864166 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmr6\" (UniqueName: \"kubernetes.io/projected/34c70bf6-91d7-4998-a593-72164375b1ee-kube-api-access-jhmr6\") pod \"test-curl-7353-140202966496576\" (UID: \"34c70bf6-91d7-4998-a593-72164375b1ee\") " pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:35.964838 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:35.964808 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmr6\" (UniqueName: \"kubernetes.io/projected/34c70bf6-91d7-4998-a593-72164375b1ee-kube-api-access-jhmr6\") pod \"test-curl-7353-140202966496576\" (UID: \"34c70bf6-91d7-4998-a593-72164375b1ee\") " pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:35.982024 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:35.981991 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmr6\" (UniqueName: \"kubernetes.io/projected/34c70bf6-91d7-4998-a593-72164375b1ee-kube-api-access-jhmr6\") pod \"test-curl-7353-140202966496576\" (UID: \"34c70bf6-91d7-4998-a593-72164375b1ee\") " pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:36.072952 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:36.072892 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:36.342531 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:36.342505 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-140202966496576"] Sep 07 04:06:36.344295 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:06:36.344265 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34c70bf6_91d7_4998_a593_72164375b1ee.slice/crio-e499ac94300746b88fc978929cd7b3923b47aa04818fc945cca07bec10c07524 WatchSource:0}: Error finding container e499ac94300746b88fc978929cd7b3923b47aa04818fc945cca07bec10c07524: Status 404 returned error can't find the container with id e499ac94300746b88fc978929cd7b3923b47aa04818fc945cca07bec10c07524 Sep 07 04:06:36.674750 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:36.674670 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-140202966496576" event={"ID":"34c70bf6-91d7-4998-a593-72164375b1ee","Type":"ContainerStarted","Data":"e499ac94300746b88fc978929cd7b3923b47aa04818fc945cca07bec10c07524"} Sep 07 04:06:37.681688 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:37.681654 2555 generic.go:356] "Generic (PLEG): container finished" podID="34c70bf6-91d7-4998-a593-72164375b1ee" containerID="6da848b64030a62bbe2f1edc98ab8f4b6c7eb65149fb29c223ada7e1de80e332" exitCode=0 Sep 07 04:06:37.682085 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:37.681722 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-140202966496576" event={"ID":"34c70bf6-91d7-4998-a593-72164375b1ee","Type":"ContainerDied","Data":"6da848b64030a62bbe2f1edc98ab8f4b6c7eb65149fb29c223ada7e1de80e332"} Sep 07 04:06:38.821888 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.821859 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:38.853942 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.853912 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7353-140202966496576_34c70bf6-91d7-4998-a593-72164375b1ee/test-curl-7353-140202966496576/0.log" Sep 07 04:06:38.876699 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.876672 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7353-140202966496576"] Sep 07 04:06:38.878626 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.878605 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7353-140202966496576"] Sep 07 04:06:38.891850 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.891828 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhmr6\" (UniqueName: \"kubernetes.io/projected/34c70bf6-91d7-4998-a593-72164375b1ee-kube-api-access-jhmr6\") pod \"34c70bf6-91d7-4998-a593-72164375b1ee\" (UID: \"34c70bf6-91d7-4998-a593-72164375b1ee\") " Sep 07 04:06:38.893818 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.893791 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c70bf6-91d7-4998-a593-72164375b1ee-kube-api-access-jhmr6" (OuterVolumeSpecName: "kube-api-access-jhmr6") pod "34c70bf6-91d7-4998-a593-72164375b1ee" (UID: "34c70bf6-91d7-4998-a593-72164375b1ee"). InnerVolumeSpecName "kube-api-access-jhmr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:06:38.989720 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.989686 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7353-140202966496576"] Sep 07 04:06:38.990106 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.990092 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c70bf6-91d7-4998-a593-72164375b1ee" containerName="test-curl-7353-140202966496576" Sep 07 04:06:38.990160 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.990108 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c70bf6-91d7-4998-a593-72164375b1ee" containerName="test-curl-7353-140202966496576" Sep 07 04:06:38.990194 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.990174 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c70bf6-91d7-4998-a593-72164375b1ee" containerName="test-curl-7353-140202966496576" Sep 07 04:06:38.993134 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.993111 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmr6\" (UniqueName: \"kubernetes.io/projected/34c70bf6-91d7-4998-a593-72164375b1ee-kube-api-access-jhmr6\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:06:38.993237 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.993224 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:38.999573 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:38.999550 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-140202966496576"] Sep 07 04:06:39.094590 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:39.094555 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s22tv\" (UniqueName: \"kubernetes.io/projected/56ac3bc5-fa48-4449-808a-49fac1d20cc3-kube-api-access-s22tv\") pod \"test-curl-7353-140202966496576\" (UID: \"56ac3bc5-fa48-4449-808a-49fac1d20cc3\") " pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:39.195991 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:39.195965 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s22tv\" (UniqueName: \"kubernetes.io/projected/56ac3bc5-fa48-4449-808a-49fac1d20cc3-kube-api-access-s22tv\") pod \"test-curl-7353-140202966496576\" (UID: \"56ac3bc5-fa48-4449-808a-49fac1d20cc3\") " pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:39.204305 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:39.204279 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s22tv\" (UniqueName: \"kubernetes.io/projected/56ac3bc5-fa48-4449-808a-49fac1d20cc3-kube-api-access-s22tv\") pod \"test-curl-7353-140202966496576\" (UID: \"56ac3bc5-fa48-4449-808a-49fac1d20cc3\") " pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:39.303712 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:39.303633 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:39.572033 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:39.572008 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-140202966496576"] Sep 07 04:06:39.574481 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:06:39.574450 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56ac3bc5_fa48_4449_808a_49fac1d20cc3.slice/crio-167c677f25aed1d8fb9d5bdfb12b69555c945c10b1832eb0987bd30bcb9424e3 WatchSource:0}: Error finding container 167c677f25aed1d8fb9d5bdfb12b69555c945c10b1832eb0987bd30bcb9424e3: Status 404 returned error can't find the container with id 167c677f25aed1d8fb9d5bdfb12b69555c945c10b1832eb0987bd30bcb9424e3 Sep 07 04:06:39.693020 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:39.692986 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-140202966496576" event={"ID":"56ac3bc5-fa48-4449-808a-49fac1d20cc3","Type":"ContainerStarted","Data":"167c677f25aed1d8fb9d5bdfb12b69555c945c10b1832eb0987bd30bcb9424e3"} Sep 07 04:06:39.694265 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:39.694242 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e499ac94300746b88fc978929cd7b3923b47aa04818fc945cca07bec10c07524" Sep 07 04:06:39.694372 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:39.694277 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:39.705731 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:39.705700 2555 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7353-140202966496576" oldPodUID="34c70bf6-91d7-4998-a593-72164375b1ee" podUID="56ac3bc5-fa48-4449-808a-49fac1d20cc3" Sep 07 04:06:39.937968 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:39.937881 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34c70bf6-91d7-4998-a593-72164375b1ee" path="/var/lib/kubelet/pods/34c70bf6-91d7-4998-a593-72164375b1ee/volumes" Sep 07 04:06:40.703333 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:40.703299 2555 generic.go:356] "Generic (PLEG): container finished" podID="56ac3bc5-fa48-4449-808a-49fac1d20cc3" containerID="f8b94a9c4581f09be0ff9f355a7b7e56a8cbe862e92f3e9fefffe0541a026ce7" exitCode=0 Sep 07 04:06:40.703504 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:40.703345 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-140202966496576" event={"ID":"56ac3bc5-fa48-4449-808a-49fac1d20cc3","Type":"ContainerDied","Data":"f8b94a9c4581f09be0ff9f355a7b7e56a8cbe862e92f3e9fefffe0541a026ce7"} Sep 07 04:06:41.845706 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:41.845679 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:41.864630 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:41.864591 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7353-140202966496576_56ac3bc5-fa48-4449-808a-49fac1d20cc3/test-curl-7353-140202966496576/0.log" Sep 07 04:06:41.889154 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:41.889128 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7353-140202966496576"] Sep 07 04:06:41.892037 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:41.892015 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7353-140202966496576"] Sep 07 04:06:41.921808 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:41.921784 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s22tv\" (UniqueName: \"kubernetes.io/projected/56ac3bc5-fa48-4449-808a-49fac1d20cc3-kube-api-access-s22tv\") pod \"56ac3bc5-fa48-4449-808a-49fac1d20cc3\" (UID: \"56ac3bc5-fa48-4449-808a-49fac1d20cc3\") " Sep 07 04:06:41.923794 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:41.923762 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ac3bc5-fa48-4449-808a-49fac1d20cc3-kube-api-access-s22tv" (OuterVolumeSpecName: "kube-api-access-s22tv") pod "56ac3bc5-fa48-4449-808a-49fac1d20cc3" (UID: "56ac3bc5-fa48-4449-808a-49fac1d20cc3"). InnerVolumeSpecName "kube-api-access-s22tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:06:41.937707 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:41.937676 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ac3bc5-fa48-4449-808a-49fac1d20cc3" path="/var/lib/kubelet/pods/56ac3bc5-fa48-4449-808a-49fac1d20cc3/volumes" Sep 07 04:06:42.003465 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.003386 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7353-140202966497152"] Sep 07 04:06:42.003796 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.003783 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ac3bc5-fa48-4449-808a-49fac1d20cc3" containerName="test-curl-7353-140202966496576" Sep 07 04:06:42.003844 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.003798 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ac3bc5-fa48-4449-808a-49fac1d20cc3" containerName="test-curl-7353-140202966496576" Sep 07 04:06:42.003880 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.003852 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ac3bc5-fa48-4449-808a-49fac1d20cc3" containerName="test-curl-7353-140202966496576" Sep 07 04:06:42.006853 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.006837 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:42.010523 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.010497 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-140202966497152"] Sep 07 04:06:42.022436 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.022406 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s22tv\" (UniqueName: \"kubernetes.io/projected/56ac3bc5-fa48-4449-808a-49fac1d20cc3-kube-api-access-s22tv\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:06:42.122839 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.122807 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v92s4\" (UniqueName: \"kubernetes.io/projected/90f64f34-2c86-498a-9f91-284e3ba47e05-kube-api-access-v92s4\") pod \"test-curl-7353-140202966497152\" (UID: \"90f64f34-2c86-498a-9f91-284e3ba47e05\") " pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:42.223710 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.223672 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v92s4\" (UniqueName: \"kubernetes.io/projected/90f64f34-2c86-498a-9f91-284e3ba47e05-kube-api-access-v92s4\") pod \"test-curl-7353-140202966497152\" (UID: \"90f64f34-2c86-498a-9f91-284e3ba47e05\") " pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:42.231352 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.231322 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v92s4\" (UniqueName: \"kubernetes.io/projected/90f64f34-2c86-498a-9f91-284e3ba47e05-kube-api-access-v92s4\") pod \"test-curl-7353-140202966497152\" (UID: \"90f64f34-2c86-498a-9f91-284e3ba47e05\") " pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:42.318390 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.318309 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:42.585828 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.585806 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-140202966497152"] Sep 07 04:06:42.587891 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:06:42.587868 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90f64f34_2c86_498a_9f91_284e3ba47e05.slice/crio-5fe4104ba0e558446bcbff6818e233abab6f355b1f33ee2c7062d8e7146acccf WatchSource:0}: Error finding container 5fe4104ba0e558446bcbff6818e233abab6f355b1f33ee2c7062d8e7146acccf: Status 404 returned error can't find the container with id 5fe4104ba0e558446bcbff6818e233abab6f355b1f33ee2c7062d8e7146acccf Sep 07 04:06:42.716189 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.716158 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966496576" Sep 07 04:06:42.716189 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.716182 2555 scope.go:117] "RemoveContainer" containerID="f8b94a9c4581f09be0ff9f355a7b7e56a8cbe862e92f3e9fefffe0541a026ce7" Sep 07 04:06:42.717674 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:42.717646 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-140202966497152" event={"ID":"90f64f34-2c86-498a-9f91-284e3ba47e05","Type":"ContainerStarted","Data":"5fe4104ba0e558446bcbff6818e233abab6f355b1f33ee2c7062d8e7146acccf"} Sep 07 04:06:43.725322 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:43.725288 2555 generic.go:356] "Generic (PLEG): container finished" podID="90f64f34-2c86-498a-9f91-284e3ba47e05" containerID="30f5eb9c78dcca4db7170a2fc454ac56cc7de700dd9fbef19557617e31920054" exitCode=0 Sep 07 04:06:43.725731 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:43.725378 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-140202966497152" event={"ID":"90f64f34-2c86-498a-9f91-284e3ba47e05","Type":"ContainerDied","Data":"30f5eb9c78dcca4db7170a2fc454ac56cc7de700dd9fbef19557617e31920054"} Sep 07 04:06:44.865702 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:44.865680 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:44.885032 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:44.885003 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7353-140202966497152_90f64f34-2c86-498a-9f91-284e3ba47e05/test-curl-7353-140202966497152/0.log" Sep 07 04:06:44.909992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:44.909962 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7353-140202966497152"] Sep 07 04:06:44.912700 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:44.912676 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7353-140202966497152"] Sep 07 04:06:44.947893 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:44.947859 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v92s4\" (UniqueName: \"kubernetes.io/projected/90f64f34-2c86-498a-9f91-284e3ba47e05-kube-api-access-v92s4\") pod \"90f64f34-2c86-498a-9f91-284e3ba47e05\" (UID: \"90f64f34-2c86-498a-9f91-284e3ba47e05\") " Sep 07 04:06:44.949845 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:44.949817 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f64f34-2c86-498a-9f91-284e3ba47e05-kube-api-access-v92s4" (OuterVolumeSpecName: "kube-api-access-v92s4") pod "90f64f34-2c86-498a-9f91-284e3ba47e05" (UID: "90f64f34-2c86-498a-9f91-284e3ba47e05"). InnerVolumeSpecName "kube-api-access-v92s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:06:45.029816 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.029735 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7353-140202966497152"] Sep 07 04:06:45.030173 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.030158 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f64f34-2c86-498a-9f91-284e3ba47e05" containerName="test-curl-7353-140202966497152" Sep 07 04:06:45.030173 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.030174 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f64f34-2c86-498a-9f91-284e3ba47e05" containerName="test-curl-7353-140202966497152" Sep 07 04:06:45.030260 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.030235 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f64f34-2c86-498a-9f91-284e3ba47e05" containerName="test-curl-7353-140202966497152" Sep 07 04:06:45.033307 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.033281 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:45.039983 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.039957 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-140202966497152"] Sep 07 04:06:45.049709 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.049685 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v92s4\" (UniqueName: \"kubernetes.io/projected/90f64f34-2c86-498a-9f91-284e3ba47e05-kube-api-access-v92s4\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:06:45.150302 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.150266 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzn54\" (UniqueName: \"kubernetes.io/projected/3c6902e8-5de6-40b9-991a-790f155dda0d-kube-api-access-hzn54\") pod \"test-curl-7353-140202966497152\" (UID: \"3c6902e8-5de6-40b9-991a-790f155dda0d\") " pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:45.251454 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.251420 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzn54\" (UniqueName: \"kubernetes.io/projected/3c6902e8-5de6-40b9-991a-790f155dda0d-kube-api-access-hzn54\") pod \"test-curl-7353-140202966497152\" (UID: \"3c6902e8-5de6-40b9-991a-790f155dda0d\") " pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:45.259405 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.259373 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzn54\" (UniqueName: \"kubernetes.io/projected/3c6902e8-5de6-40b9-991a-790f155dda0d-kube-api-access-hzn54\") pod \"test-curl-7353-140202966497152\" (UID: \"3c6902e8-5de6-40b9-991a-790f155dda0d\") " pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:45.345625 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.345536 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:45.615949 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.615922 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-140202966497152"] Sep 07 04:06:45.618570 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:06:45.618547 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c6902e8_5de6_40b9_991a_790f155dda0d.slice/crio-b0f3bfe69021a94a52bfa35ba58e6da89e37751870a1e3d1bd2c251baeb1046b WatchSource:0}: Error finding container b0f3bfe69021a94a52bfa35ba58e6da89e37751870a1e3d1bd2c251baeb1046b: Status 404 returned error can't find the container with id b0f3bfe69021a94a52bfa35ba58e6da89e37751870a1e3d1bd2c251baeb1046b Sep 07 04:06:45.738496 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.738451 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-140202966497152" event={"ID":"3c6902e8-5de6-40b9-991a-790f155dda0d","Type":"ContainerStarted","Data":"b0f3bfe69021a94a52bfa35ba58e6da89e37751870a1e3d1bd2c251baeb1046b"} Sep 07 04:06:45.739919 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.739892 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:45.739919 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.739915 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fe4104ba0e558446bcbff6818e233abab6f355b1f33ee2c7062d8e7146acccf" Sep 07 04:06:45.751329 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.751301 2555 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7353-140202966497152" oldPodUID="90f64f34-2c86-498a-9f91-284e3ba47e05" podUID="3c6902e8-5de6-40b9-991a-790f155dda0d" Sep 07 04:06:45.938763 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:45.938677 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f64f34-2c86-498a-9f91-284e3ba47e05" path="/var/lib/kubelet/pods/90f64f34-2c86-498a-9f91-284e3ba47e05/volumes" Sep 07 04:06:46.194951 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:46.194919 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:06:46.747254 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:46.747219 2555 generic.go:356] "Generic (PLEG): container finished" podID="3c6902e8-5de6-40b9-991a-790f155dda0d" containerID="73a9a24ca01e12762a7122c466d738daeb5c43ec344cf3fe4937c28d6adbef07" exitCode=0 Sep 07 04:06:46.747435 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:46.747315 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-140202966497152" event={"ID":"3c6902e8-5de6-40b9-991a-790f155dda0d","Type":"ContainerDied","Data":"73a9a24ca01e12762a7122c466d738daeb5c43ec344cf3fe4937c28d6adbef07"} Sep 07 04:06:47.889220 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:47.889197 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:47.908241 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:47.908196 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7353-140202966497152_3c6902e8-5de6-40b9-991a-790f155dda0d/test-curl-7353-140202966497152/0.log" Sep 07 04:06:47.938661 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:47.938632 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7353-140202966497152"] Sep 07 04:06:47.938811 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:47.938675 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7353-140202966497152"] Sep 07 04:06:47.978466 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:47.978429 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzn54\" (UniqueName: \"kubernetes.io/projected/3c6902e8-5de6-40b9-991a-790f155dda0d-kube-api-access-hzn54\") pod \"3c6902e8-5de6-40b9-991a-790f155dda0d\" (UID: \"3c6902e8-5de6-40b9-991a-790f155dda0d\") " Sep 07 04:06:47.980530 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:47.980494 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6902e8-5de6-40b9-991a-790f155dda0d-kube-api-access-hzn54" (OuterVolumeSpecName: "kube-api-access-hzn54") pod "3c6902e8-5de6-40b9-991a-790f155dda0d" (UID: "3c6902e8-5de6-40b9-991a-790f155dda0d"). InnerVolumeSpecName "kube-api-access-hzn54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:06:48.079892 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.079803 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzn54\" (UniqueName: \"kubernetes.io/projected/3c6902e8-5de6-40b9-991a-790f155dda0d-kube-api-access-hzn54\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:06:48.758987 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.758960 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:48.759177 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.759003 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0f3bfe69021a94a52bfa35ba58e6da89e37751870a1e3d1bd2c251baeb1046b" Sep 07 04:06:48.769100 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:06:48.769040 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7353-140202966497152\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="3c6902e8-5de6-40b9-991a-790f155dda0d" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:48.791026 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.790989 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k"] Sep 07 04:06:48.791675 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.791651 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c6902e8-5de6-40b9-991a-790f155dda0d" containerName="test-curl-7353-140202966497152" Sep 07 04:06:48.791675 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.791674 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c6902e8-5de6-40b9-991a-790f155dda0d" containerName="test-curl-7353-140202966497152" Sep 07 04:06:48.791876 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.791772 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c6902e8-5de6-40b9-991a-790f155dda0d" containerName="test-curl-7353-140202966497152" Sep 07 04:06:48.795193 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.795176 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.797752 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:06:48.797728 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7353-140202966497152\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="3c6902e8-5de6-40b9-991a-790f155dda0d" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:48.797875 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.797856 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-107fc16f-openshift-default-dockercfg-d4n9m\"" Sep 07 04:06:48.804622 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.804600 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k"] Sep 07 04:06:48.887590 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.887551 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967f54ea-affd-47e7-92eb-497d963768bc-istio-podinfo\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.887797 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.887668 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqs99\" (UniqueName: \"kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-kube-api-access-vqs99\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.887797 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.887698 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-envoy\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.887797 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.887772 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-credential-socket\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.887973 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.887819 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-socket\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.887973 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.887899 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-data\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.887973 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.887945 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-certs\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.888163 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.888010 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-istio-token\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.888163 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.888071 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967f54ea-affd-47e7-92eb-497d963768bc-istiod-ca-cert\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.988567 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.988526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967f54ea-affd-47e7-92eb-497d963768bc-istiod-ca-cert\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.988591 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967f54ea-affd-47e7-92eb-497d963768bc-istio-podinfo\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.988639 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqs99\" (UniqueName: \"kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-kube-api-access-vqs99\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.988667 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-envoy\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.988709 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-credential-socket\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.988741 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-socket\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.988814 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-data\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.988858 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-certs\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989046 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.988891 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-istio-token\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989451 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.989253 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-credential-socket\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989451 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.989313 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-certs\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989451 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.989329 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-data\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989451 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.989431 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967f54ea-affd-47e7-92eb-497d963768bc-istiod-ca-cert\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.989588 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.989463 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-socket\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.991269 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.991249 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-envoy\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.991642 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.991620 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967f54ea-affd-47e7-92eb-497d963768bc-istio-podinfo\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.996421 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.996393 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-istio-token\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:48.996506 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:48.996460 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqs99\" (UniqueName: \"kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-kube-api-access-vqs99\") pod \"e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:49.107749 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:49.107631 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:49.387628 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:49.387600 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k"] Sep 07 04:06:49.389509 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:06:49.389481 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967f54ea_affd_47e7_92eb_497d963768bc.slice/crio-0660a2ef15bd0fca9a1f71245592e7e4c1c18c1ff8bbcfd2c3ad7f9051d01526 WatchSource:0}: Error finding container 0660a2ef15bd0fca9a1f71245592e7e4c1c18c1ff8bbcfd2c3ad7f9051d01526: Status 404 returned error can't find the container with id 0660a2ef15bd0fca9a1f71245592e7e4c1c18c1ff8bbcfd2c3ad7f9051d01526 Sep 07 04:06:49.391840 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:49.391806 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:06:49.391930 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:49.391873 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:06:49.391930 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:49.391900 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:06:49.768404 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:49.768360 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" event={"ID":"967f54ea-affd-47e7-92eb-497d963768bc","Type":"ContainerStarted","Data":"027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3"} Sep 07 04:06:49.768666 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:49.768462 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" event={"ID":"967f54ea-affd-47e7-92eb-497d963768bc","Type":"ContainerStarted","Data":"0660a2ef15bd0fca9a1f71245592e7e4c1c18c1ff8bbcfd2c3ad7f9051d01526"} Sep 07 04:06:49.788072 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:49.788018 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" podStartSLOduration=1.7880027790000002 podStartE2EDuration="1.788002779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:06:49.786249417 +0000 UTC m=+1128.361115705" watchObservedRunningTime="2026-09-07 04:06:49.788002779 +0000 UTC m=+1128.362869070" Sep 07 04:06:49.788226 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:06:49.788200 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7353-140202966497152\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="3c6902e8-5de6-40b9-991a-790f155dda0d" pod="opendatahub/test-curl-7353-140202966497152" Sep 07 04:06:49.937801 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:49.937760 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c6902e8-5de6-40b9-991a-790f155dda0d" path="/var/lib/kubelet/pods/3c6902e8-5de6-40b9-991a-790f155dda0d/volumes" Sep 07 04:06:50.107900 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:50.107816 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:50.109452 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:50.109422 2555 patch_prober.go:28] interesting pod/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 07 04:06:50.109575 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:50.109481 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 07 04:06:51.111988 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:51.111954 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:51.779997 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:51.779965 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:51.780885 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:51.780864 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:06:52.893511 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:52.893477 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7"] Sep 07 04:06:52.898616 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:52.898594 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:52.900801 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:52.900775 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-107fc16f-serving-cert\"" Sep 07 04:06:52.912630 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:52.912604 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7"] Sep 07 04:06:53.036563 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.036520 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb81431-38fe-4fee-87f3-5b24caed7191-openshift-service-ca\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.036774 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.036652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wgjn\" (UniqueName: \"kubernetes.io/projected/7bb81431-38fe-4fee-87f3-5b24caed7191-kube-api-access-4wgjn\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.036845 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.036775 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-maas-api-tls\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.036845 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.036796 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-metrics-certs\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.137943 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.137898 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgjn\" (UniqueName: \"kubernetes.io/projected/7bb81431-38fe-4fee-87f3-5b24caed7191-kube-api-access-4wgjn\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.138177 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.137957 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-maas-api-tls\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.138177 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.137975 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-metrics-certs\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.138177 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.138007 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb81431-38fe-4fee-87f3-5b24caed7191-openshift-service-ca\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.138913 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.138884 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb81431-38fe-4fee-87f3-5b24caed7191-openshift-service-ca\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.140629 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.140605 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-metrics-certs\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.140725 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.140644 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-maas-api-tls\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.146013 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.145951 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgjn\" (UniqueName: \"kubernetes.io/projected/7bb81431-38fe-4fee-87f3-5b24caed7191-kube-api-access-4wgjn\") pod \"maas-api-e2e-mt-107fc16f-596694fb95-hsbk7\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.211823 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.211786 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:53.491066 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.491029 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7"] Sep 07 04:06:53.493600 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:06:53.493575 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb81431_38fe_4fee_87f3_5b24caed7191.slice/crio-56b996f93b6e120062a0ec30249c36cded0cefe4bc6ddc1e337ee52ddb8da5d6 WatchSource:0}: Error finding container 56b996f93b6e120062a0ec30249c36cded0cefe4bc6ddc1e337ee52ddb8da5d6: Status 404 returned error can't find the container with id 56b996f93b6e120062a0ec30249c36cded0cefe4bc6ddc1e337ee52ddb8da5d6 Sep 07 04:06:53.791321 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:53.791227 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" event={"ID":"7bb81431-38fe-4fee-87f3-5b24caed7191","Type":"ContainerStarted","Data":"56b996f93b6e120062a0ec30249c36cded0cefe4bc6ddc1e337ee52ddb8da5d6"} Sep 07 04:06:54.798192 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:54.798152 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" event={"ID":"7bb81431-38fe-4fee-87f3-5b24caed7191","Type":"ContainerStarted","Data":"78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5"} Sep 07 04:06:54.798603 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:54.798284 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:06:54.815599 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:06:54.815541 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" podStartSLOduration=2.386376828 podStartE2EDuration="2.815519151s" totalImagesPullingTime="429.142323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:06:54.814445825 +0000 UTC m=+1133.389312114" watchObservedRunningTime="2026-09-07 04:06:54.815519151 +0000 UTC m=+1133.390385440" Sep 07 04:07:00.809981 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:00.809952 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:07:02.046266 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:02.046221 2555 scope.go:117] "RemoveContainer" containerID="58675bd9817179dfa2b0ea820d5420d85a9fd7b66951cee0bd77efeadefe7aae" Sep 07 04:07:02.063380 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:02.063355 2555 scope.go:117] "RemoveContainer" containerID="9d3350f404faaffc74b01b64c642bcb65ef604cb3f29d5db19dd2f237fc2163a" Sep 07 04:07:03.897595 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:03.897557 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:07:14.922923 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:14.922887 2555 generic.go:356] "Generic (PLEG): container finished" podID="967f54ea-affd-47e7-92eb-497d963768bc" containerID="027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3" exitCode=0 Sep 07 04:07:14.923369 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:14.922962 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" event={"ID":"967f54ea-affd-47e7-92eb-497d963768bc","Type":"ContainerDied","Data":"027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3"} Sep 07 04:07:14.923467 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:14.923452 2555 scope.go:117] "RemoveContainer" containerID="027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3" Sep 07 04:07:14.924589 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:14.924559 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:07:14.924698 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:14.924641 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:07:14.924698 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:14.924691 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:07:15.943737 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:15.943695 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" event={"ID":"967f54ea-affd-47e7-92eb-497d963768bc","Type":"ContainerStarted","Data":"3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941"} Sep 07 04:07:16.108089 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:16.108027 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:07:16.108318 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:16.108295 2555 patch_prober.go:28] interesting pod/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 07 04:07:16.108387 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:16.108357 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 07 04:07:17.108551 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:17.108513 2555 patch_prober.go:28] interesting pod/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 07 04:07:17.109022 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:17.108600 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 07 04:07:18.108032 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:18.107993 2555 patch_prober.go:28] interesting pod/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 07 04:07:18.108248 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:18.108091 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 07 04:07:19.108995 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:19.108960 2555 patch_prober.go:28] interesting pod/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 07 04:07:19.109482 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:19.109037 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 07 04:07:20.108579 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:20.108549 2555 patch_prober.go:28] interesting pod/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 07 04:07:20.108780 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:20.108600 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 07 04:07:21.108598 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:21.108559 2555 patch_prober.go:28] interesting pod/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 07 04:07:21.109110 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:21.108623 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 07 04:07:21.780471 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:21.780433 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:07:22.113428 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:22.113329 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:07:22.114619 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:22.114596 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:07:24.491551 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:24.491461 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:07:44.789346 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:07:44.789304 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:08:01.979234 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:01.979204 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:08:01.979678 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:01.979532 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:08:02.096870 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:02.096837 2555 scope.go:117] "RemoveContainer" containerID="21729231c8582397de6981ea55c315eb040b4022e1f56610a0a499c06a7b5a0d" Sep 07 04:08:05.287497 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:05.287459 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:08:06.342740 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.342705 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7"] Sep 07 04:08:06.343222 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.342961 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" podUID="7bb81431-38fe-4fee-87f3-5b24caed7191" containerName="maas-api" containerID="cri-o://78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5" gracePeriod=30 Sep 07 04:08:06.608981 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.608953 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:08:06.676329 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.676291 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-maas-api-tls\") pod \"7bb81431-38fe-4fee-87f3-5b24caed7191\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " Sep 07 04:08:06.676511 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.676342 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-metrics-certs\") pod \"7bb81431-38fe-4fee-87f3-5b24caed7191\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " Sep 07 04:08:06.676511 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.676402 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb81431-38fe-4fee-87f3-5b24caed7191-openshift-service-ca\") pod \"7bb81431-38fe-4fee-87f3-5b24caed7191\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " Sep 07 04:08:06.676511 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.676465 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wgjn\" (UniqueName: \"kubernetes.io/projected/7bb81431-38fe-4fee-87f3-5b24caed7191-kube-api-access-4wgjn\") pod \"7bb81431-38fe-4fee-87f3-5b24caed7191\" (UID: \"7bb81431-38fe-4fee-87f3-5b24caed7191\") " Sep 07 04:08:06.676793 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.676750 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb81431-38fe-4fee-87f3-5b24caed7191-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bb81431-38fe-4fee-87f3-5b24caed7191" (UID: "7bb81431-38fe-4fee-87f3-5b24caed7191"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:08:06.678521 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.678488 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb81431-38fe-4fee-87f3-5b24caed7191-kube-api-access-4wgjn" (OuterVolumeSpecName: "kube-api-access-4wgjn") pod "7bb81431-38fe-4fee-87f3-5b24caed7191" (UID: "7bb81431-38fe-4fee-87f3-5b24caed7191"). InnerVolumeSpecName "kube-api-access-4wgjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:08:06.678612 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.678516 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bb81431-38fe-4fee-87f3-5b24caed7191" (UID: "7bb81431-38fe-4fee-87f3-5b24caed7191"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:08:06.678612 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.678496 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7bb81431-38fe-4fee-87f3-5b24caed7191" (UID: "7bb81431-38fe-4fee-87f3-5b24caed7191"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:08:06.777501 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.777464 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wgjn\" (UniqueName: \"kubernetes.io/projected/7bb81431-38fe-4fee-87f3-5b24caed7191-kube-api-access-4wgjn\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:08:06.777501 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.777496 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-maas-api-tls\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:08:06.777501 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.777506 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb81431-38fe-4fee-87f3-5b24caed7191-metrics-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:08:06.777501 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:06.777513 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb81431-38fe-4fee-87f3-5b24caed7191-openshift-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:08:07.220224 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:07.220189 2555 generic.go:356] "Generic (PLEG): container finished" podID="7bb81431-38fe-4fee-87f3-5b24caed7191" containerID="78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5" exitCode=0 Sep 07 04:08:07.220440 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:07.220233 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" Sep 07 04:08:07.220440 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:07.220279 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" event={"ID":"7bb81431-38fe-4fee-87f3-5b24caed7191","Type":"ContainerDied","Data":"78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5"} Sep 07 04:08:07.220440 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:07.220344 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7" event={"ID":"7bb81431-38fe-4fee-87f3-5b24caed7191","Type":"ContainerDied","Data":"56b996f93b6e120062a0ec30249c36cded0cefe4bc6ddc1e337ee52ddb8da5d6"} Sep 07 04:08:07.220440 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:07.220374 2555 scope.go:117] "RemoveContainer" containerID="78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5" Sep 07 04:08:07.229940 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:07.229924 2555 scope.go:117] "RemoveContainer" containerID="78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5" Sep 07 04:08:07.230239 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:08:07.230220 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5\": container with ID starting with 78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5 not found: ID does not exist" containerID="78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5" Sep 07 04:08:07.230289 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:07.230247 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5"} err="failed to get container status \"78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5\": rpc error: code = NotFound desc = could not find container \"78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5\": container with ID starting with 78e2076c65b719505dd498c0ceec42d2858c99ed8d97780b86bb5b2ac8b2b5d5 not found: ID does not exist" Sep 07 04:08:07.246835 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:07.246794 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7"] Sep 07 04:08:07.247003 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:07.246847 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-107fc16f-596694fb95-hsbk7"] Sep 07 04:08:07.937647 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:07.937606 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb81431-38fe-4fee-87f3-5b24caed7191" path="/var/lib/kubelet/pods/7bb81431-38fe-4fee-87f3-5b24caed7191/volumes" Sep 07 04:08:24.188259 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:24.188218 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:08:40.785962 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:40.785920 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:08:51.104627 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.104591 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7356-744c78"] Sep 07 04:08:51.105108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.104986 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb81431-38fe-4fee-87f3-5b24caed7191" containerName="maas-api" Sep 07 04:08:51.105108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.104998 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb81431-38fe-4fee-87f3-5b24caed7191" containerName="maas-api" Sep 07 04:08:51.105108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.105090 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb81431-38fe-4fee-87f3-5b24caed7191" containerName="maas-api" Sep 07 04:08:51.108333 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.108316 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-744c78" Sep 07 04:08:51.111282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.111257 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhk6n\"" Sep 07 04:08:51.113529 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.113503 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7356-744c78"] Sep 07 04:08:51.205107 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.205070 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl499\" (UniqueName: \"kubernetes.io/projected/22554e6e-f3a9-4b94-8337-d428d2203924-kube-api-access-tl499\") pod \"mt-curl-7356-744c78\" (UID: \"22554e6e-f3a9-4b94-8337-d428d2203924\") " pod="opendatahub/mt-curl-7356-744c78" Sep 07 04:08:51.305992 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.305958 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl499\" (UniqueName: \"kubernetes.io/projected/22554e6e-f3a9-4b94-8337-d428d2203924-kube-api-access-tl499\") pod \"mt-curl-7356-744c78\" (UID: \"22554e6e-f3a9-4b94-8337-d428d2203924\") " pod="opendatahub/mt-curl-7356-744c78" Sep 07 04:08:51.315262 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.315233 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl499\" (UniqueName: \"kubernetes.io/projected/22554e6e-f3a9-4b94-8337-d428d2203924-kube-api-access-tl499\") pod \"mt-curl-7356-744c78\" (UID: \"22554e6e-f3a9-4b94-8337-d428d2203924\") " pod="opendatahub/mt-curl-7356-744c78" Sep 07 04:08:51.419478 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.419386 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-744c78" Sep 07 04:08:51.689919 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:51.689843 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7356-744c78"] Sep 07 04:08:51.692696 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:08:51.692670 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22554e6e_f3a9_4b94_8337_d428d2203924.slice/crio-38480de3399057450e2403b33f58d9da1f21cbd281d3ce9c83550390080484bc WatchSource:0}: Error finding container 38480de3399057450e2403b33f58d9da1f21cbd281d3ce9c83550390080484bc: Status 404 returned error can't find the container with id 38480de3399057450e2403b33f58d9da1f21cbd281d3ce9c83550390080484bc Sep 07 04:08:52.470112 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:52.470085 2555 generic.go:356] "Generic (PLEG): container finished" podID="22554e6e-f3a9-4b94-8337-d428d2203924" containerID="2b8edbb380dbf1874769ad80c9ac9723b5c56d5adc62e06159bd7dbb7284ce84" exitCode=0 Sep 07 04:08:52.470398 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:52.470217 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7356-744c78" event={"ID":"22554e6e-f3a9-4b94-8337-d428d2203924","Type":"ContainerDied","Data":"2b8edbb380dbf1874769ad80c9ac9723b5c56d5adc62e06159bd7dbb7284ce84"} Sep 07 04:08:52.470398 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:52.470256 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7356-744c78" event={"ID":"22554e6e-f3a9-4b94-8337-d428d2203924","Type":"ContainerStarted","Data":"38480de3399057450e2403b33f58d9da1f21cbd281d3ce9c83550390080484bc"} Sep 07 04:08:53.622325 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:53.622302 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-744c78" Sep 07 04:08:53.651334 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:53.651305 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7356-744c78_22554e6e-f3a9-4b94-8337-d428d2203924/mt-curl-7356-744c78/0.log" Sep 07 04:08:53.673339 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:53.673315 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7356-744c78"] Sep 07 04:08:53.676883 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:53.676860 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7356-744c78"] Sep 07 04:08:53.728404 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:53.728379 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl499\" (UniqueName: \"kubernetes.io/projected/22554e6e-f3a9-4b94-8337-d428d2203924-kube-api-access-tl499\") pod \"22554e6e-f3a9-4b94-8337-d428d2203924\" (UID: \"22554e6e-f3a9-4b94-8337-d428d2203924\") " Sep 07 04:08:53.730348 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:53.730318 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22554e6e-f3a9-4b94-8337-d428d2203924-kube-api-access-tl499" (OuterVolumeSpecName: "kube-api-access-tl499") pod "22554e6e-f3a9-4b94-8337-d428d2203924" (UID: "22554e6e-f3a9-4b94-8337-d428d2203924"). InnerVolumeSpecName "kube-api-access-tl499". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:08:53.829879 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:53.829852 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl499\" (UniqueName: \"kubernetes.io/projected/22554e6e-f3a9-4b94-8337-d428d2203924-kube-api-access-tl499\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:08:53.937899 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:53.937856 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22554e6e-f3a9-4b94-8337-d428d2203924" path="/var/lib/kubelet/pods/22554e6e-f3a9-4b94-8337-d428d2203924/volumes" Sep 07 04:08:54.483639 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:54.483609 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-744c78" Sep 07 04:08:54.483815 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:54.483617 2555 scope.go:117] "RemoveContainer" containerID="2b8edbb380dbf1874769ad80c9ac9723b5c56d5adc62e06159bd7dbb7284ce84" Sep 07 04:08:56.686127 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:56.686093 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:08:57.765800 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:57.765763 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k"] Sep 07 04:08:57.766283 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:08:57.765987 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" containerID="cri-o://3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941" gracePeriod=30 Sep 07 04:09:03.017250 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.017223 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:09:03.118762 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.118678 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967f54ea-affd-47e7-92eb-497d963768bc-istiod-ca-cert\") pod \"967f54ea-affd-47e7-92eb-497d963768bc\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " Sep 07 04:09:03.118762 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.118755 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-envoy\") pod \"967f54ea-affd-47e7-92eb-497d963768bc\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " Sep 07 04:09:03.118963 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.118773 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-istio-token\") pod \"967f54ea-affd-47e7-92eb-497d963768bc\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " Sep 07 04:09:03.118963 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.118938 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-credential-socket\") pod \"967f54ea-affd-47e7-92eb-497d963768bc\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " Sep 07 04:09:03.119094 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.118980 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-data\") pod \"967f54ea-affd-47e7-92eb-497d963768bc\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " Sep 07 04:09:03.119094 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119005 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-certs\") pod \"967f54ea-affd-47e7-92eb-497d963768bc\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " Sep 07 04:09:03.119094 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119052 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967f54ea-affd-47e7-92eb-497d963768bc-istio-podinfo\") pod \"967f54ea-affd-47e7-92eb-497d963768bc\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " Sep 07 04:09:03.119255 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119105 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967f54ea-affd-47e7-92eb-497d963768bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "967f54ea-affd-47e7-92eb-497d963768bc" (UID: "967f54ea-affd-47e7-92eb-497d963768bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:09:03.119255 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119136 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqs99\" (UniqueName: \"kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-kube-api-access-vqs99\") pod \"967f54ea-affd-47e7-92eb-497d963768bc\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " Sep 07 04:09:03.119255 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119176 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-socket\") pod \"967f54ea-affd-47e7-92eb-497d963768bc\" (UID: \"967f54ea-affd-47e7-92eb-497d963768bc\") " Sep 07 04:09:03.119384 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119220 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "967f54ea-affd-47e7-92eb-497d963768bc" (UID: "967f54ea-affd-47e7-92eb-497d963768bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:09:03.119384 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119308 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "967f54ea-affd-47e7-92eb-497d963768bc" (UID: "967f54ea-affd-47e7-92eb-497d963768bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:09:03.119384 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119332 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "967f54ea-affd-47e7-92eb-497d963768bc" (UID: "967f54ea-affd-47e7-92eb-497d963768bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:09:03.119541 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119491 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "967f54ea-affd-47e7-92eb-497d963768bc" (UID: "967f54ea-affd-47e7-92eb-497d963768bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:09:03.119658 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119639 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:09:03.119748 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119664 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:09:03.119748 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119682 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:09:03.119748 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119697 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:09:03.119748 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.119713 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967f54ea-affd-47e7-92eb-497d963768bc-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:09:03.121001 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.120973 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "967f54ea-affd-47e7-92eb-497d963768bc" (UID: "967f54ea-affd-47e7-92eb-497d963768bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:09:03.121151 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.121133 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "967f54ea-affd-47e7-92eb-497d963768bc" (UID: "967f54ea-affd-47e7-92eb-497d963768bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:09:03.121247 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.121233 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/967f54ea-affd-47e7-92eb-497d963768bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "967f54ea-affd-47e7-92eb-497d963768bc" (UID: "967f54ea-affd-47e7-92eb-497d963768bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:09:03.121286 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.121239 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-kube-api-access-vqs99" (OuterVolumeSpecName: "kube-api-access-vqs99") pod "967f54ea-affd-47e7-92eb-497d963768bc" (UID: "967f54ea-affd-47e7-92eb-497d963768bc"). InnerVolumeSpecName "kube-api-access-vqs99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:09:03.220517 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.220480 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967f54ea-affd-47e7-92eb-497d963768bc-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:09:03.220517 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.220514 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqs99\" (UniqueName: \"kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-kube-api-access-vqs99\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:09:03.220517 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.220525 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967f54ea-affd-47e7-92eb-497d963768bc-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:09:03.220760 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.220534 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967f54ea-affd-47e7-92eb-497d963768bc-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:09:03.531749 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.531714 2555 generic.go:356] "Generic (PLEG): container finished" podID="967f54ea-affd-47e7-92eb-497d963768bc" containerID="3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941" exitCode=0 Sep 07 04:09:03.531935 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.531761 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" Sep 07 04:09:03.531935 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.531789 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" event={"ID":"967f54ea-affd-47e7-92eb-497d963768bc","Type":"ContainerDied","Data":"3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941"} Sep 07 04:09:03.531935 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.531832 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k" event={"ID":"967f54ea-affd-47e7-92eb-497d963768bc","Type":"ContainerDied","Data":"0660a2ef15bd0fca9a1f71245592e7e4c1c18c1ff8bbcfd2c3ad7f9051d01526"} Sep 07 04:09:03.531935 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.531856 2555 scope.go:117] "RemoveContainer" containerID="3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941" Sep 07 04:09:03.542337 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.542316 2555 scope.go:117] "RemoveContainer" containerID="027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3" Sep 07 04:09:03.550619 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.550600 2555 scope.go:117] "RemoveContainer" containerID="3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941" Sep 07 04:09:03.550889 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:09:03.550872 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941\": container with ID starting with 3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941 not found: ID does not exist" containerID="3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941" Sep 07 04:09:03.550937 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.550898 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941"} err="failed to get container status \"3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941\": rpc error: code = NotFound desc = could not find container \"3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941\": container with ID starting with 3c4ee0f6ec48e2752de32b3fe2a4c071bc6c3b420c7d51cdb880ec088e9c7941 not found: ID does not exist" Sep 07 04:09:03.550937 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.550916 2555 scope.go:117] "RemoveContainer" containerID="027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3" Sep 07 04:09:03.551147 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:09:03.551133 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3\": container with ID starting with 027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3 not found: ID does not exist" containerID="027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3" Sep 07 04:09:03.551192 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.551150 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3"} err="failed to get container status \"027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3\": rpc error: code = NotFound desc = could not find container \"027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3\": container with ID starting with 027f3319f837ae6b5aa149653dcff5de12b42f886aa50cfd59caa03c909c35e3 not found: ID does not exist" Sep 07 04:09:03.554367 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.554347 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k"] Sep 07 04:09:03.558117 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.558098 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-107fc16f-openshift-default-7cf76cbc64-cwv8k"] Sep 07 04:09:03.937372 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:03.937285 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="967f54ea-affd-47e7-92eb-497d963768bc" path="/var/lib/kubelet/pods/967f54ea-affd-47e7-92eb-497d963768bc/volumes" Sep 07 04:09:10.896651 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:10.896604 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:09:33.198022 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:33.197976 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:09:51.649745 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.649705 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht"] Sep 07 04:09:51.650368 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.650302 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" Sep 07 04:09:51.650368 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.650320 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" Sep 07 04:09:51.650368 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.650348 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22554e6e-f3a9-4b94-8337-d428d2203924" containerName="mt-curl-7356-744c78" Sep 07 04:09:51.650368 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.650357 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="22554e6e-f3a9-4b94-8337-d428d2203924" containerName="mt-curl-7356-744c78" Sep 07 04:09:51.650368 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.650369 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" Sep 07 04:09:51.650672 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.650377 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" Sep 07 04:09:51.650672 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.650492 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" Sep 07 04:09:51.650672 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.650506 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="22554e6e-f3a9-4b94-8337-d428d2203924" containerName="mt-curl-7356-744c78" Sep 07 04:09:51.650672 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.650517 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="967f54ea-affd-47e7-92eb-497d963768bc" containerName="istio-proxy" Sep 07 04:09:51.654143 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.654124 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.656795 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.656775 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-467e44-openshift-default-dockercfg-fz4tx\"" Sep 07 04:09:51.662804 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.662526 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht"] Sep 07 04:09:51.772864 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.772827 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkgp5\" (UniqueName: \"kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-kube-api-access-tkgp5\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.772864 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.772870 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-istio-token\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.773115 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.772899 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-envoy\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.773115 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.772942 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-credential-socket\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.773115 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.772990 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-data\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.773115 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.773007 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a28179-10cf-44db-94f2-c4b06d591074-istiod-ca-cert\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.773115 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.773025 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a28179-10cf-44db-94f2-c4b06d591074-istio-podinfo\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.773304 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.773109 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-socket\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.773304 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.773169 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-certs\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874030 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874001 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkgp5\" (UniqueName: \"kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-kube-api-access-tkgp5\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874037 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-istio-token\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-envoy\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874098 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-credential-socket\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874144 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-data\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874162 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a28179-10cf-44db-94f2-c4b06d591074-istiod-ca-cert\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874187 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a28179-10cf-44db-94f2-c4b06d591074-istio-podinfo\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874217 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-socket\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874623 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874288 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-certs\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874623 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874558 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-credential-socket\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874623 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874589 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-data\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874763 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874636 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-socket\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874763 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874708 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-certs\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.874964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.874936 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a28179-10cf-44db-94f2-c4b06d591074-istiod-ca-cert\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.876699 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.876673 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-envoy\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.876777 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.876732 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a28179-10cf-44db-94f2-c4b06d591074-istio-podinfo\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.889082 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.889032 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkgp5\" (UniqueName: \"kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-kube-api-access-tkgp5\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.889208 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.889190 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-istio-token\") pod \"e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:51.967262 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:51.967238 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:52.241898 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:52.241876 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht"] Sep 07 04:09:52.244574 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:09:52.244545 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a28179_10cf_44db_94f2_c4b06d591074.slice/crio-e57b42f1c5981ced6e9c0810d862e9fc20d82714f7199d44a68fb9ce469ba65a WatchSource:0}: Error finding container e57b42f1c5981ced6e9c0810d862e9fc20d82714f7199d44a68fb9ce469ba65a: Status 404 returned error can't find the container with id e57b42f1c5981ced6e9c0810d862e9fc20d82714f7199d44a68fb9ce469ba65a Sep 07 04:09:52.247274 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:52.247233 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:09:52.247395 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:52.247327 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:09:52.247395 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:52.247372 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:09:52.796030 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:52.795988 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" event={"ID":"b8a28179-10cf-44db-94f2-c4b06d591074","Type":"ContainerStarted","Data":"36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c"} Sep 07 04:09:52.796563 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:52.796051 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" event={"ID":"b8a28179-10cf-44db-94f2-c4b06d591074","Type":"ContainerStarted","Data":"e57b42f1c5981ced6e9c0810d862e9fc20d82714f7199d44a68fb9ce469ba65a"} Sep 07 04:09:52.821032 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:52.820975 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" podStartSLOduration=1.820959372 podStartE2EDuration="1.820959372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:09:52.819167222 +0000 UTC m=+1311.394033511" watchObservedRunningTime="2026-09-07 04:09:52.820959372 +0000 UTC m=+1311.395825660" Sep 07 04:09:52.967952 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:52.967916 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:53.972981 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:53.972948 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:54.807079 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:54.807020 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:54.808444 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:54.808421 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:09:55.975704 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:55.975673 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v"] Sep 07 04:09:55.979468 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:55.979443 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:55.981861 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:55.981835 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-467e44-serving-cert\"" Sep 07 04:09:55.993384 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:55.993358 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v"] Sep 07 04:09:56.115450 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.115406 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cb2d92-20ba-48ee-9f43-b91573487691-openshift-service-ca\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.115668 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.115480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-metrics-certs\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.115668 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.115560 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-maas-api-tls\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.115668 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.115591 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9gxk\" (UniqueName: \"kubernetes.io/projected/44cb2d92-20ba-48ee-9f43-b91573487691-kube-api-access-p9gxk\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.216356 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.216308 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-metrics-certs\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.216356 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.216366 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-maas-api-tls\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.216634 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.216389 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9gxk\" (UniqueName: \"kubernetes.io/projected/44cb2d92-20ba-48ee-9f43-b91573487691-kube-api-access-p9gxk\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.216634 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.216466 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cb2d92-20ba-48ee-9f43-b91573487691-openshift-service-ca\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.216634 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:09:56.216468 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-serving-cert: secret "maas-api-e2e-api-a-467e44-serving-cert" not found Sep 07 04:09:56.216634 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:09:56.216599 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-maas-api-tls podName:44cb2d92-20ba-48ee-9f43-b91573487691 nodeName:}" failed. No retries permitted until 2026-09-07 04:09:56.716578164 +0000 UTC m=+1315.291444440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-maas-api-tls") pod "maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" (UID: "44cb2d92-20ba-48ee-9f43-b91573487691") : secret "maas-api-e2e-api-a-467e44-serving-cert" not found Sep 07 04:09:56.217108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.217084 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cb2d92-20ba-48ee-9f43-b91573487691-openshift-service-ca\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.218784 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.218760 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-metrics-certs\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.224630 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.224604 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9gxk\" (UniqueName: \"kubernetes.io/projected/44cb2d92-20ba-48ee-9f43-b91573487691-kube-api-access-p9gxk\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.721855 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.721814 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-maas-api-tls\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.724238 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.724218 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-maas-api-tls\") pod \"maas-api-e2e-api-a-467e44-7f5d95487b-lx45v\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:56.891538 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:56.891500 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:57.171096 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:57.171068 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v"] Sep 07 04:09:57.172851 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:09:57.172814 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cb2d92_20ba_48ee_9f43_b91573487691.slice/crio-d6f9a4546956942d5a39472014e0f822af9ac67dfc62d9c146b6163752fb24f4 WatchSource:0}: Error finding container d6f9a4546956942d5a39472014e0f822af9ac67dfc62d9c146b6163752fb24f4: Status 404 returned error can't find the container with id d6f9a4546956942d5a39472014e0f822af9ac67dfc62d9c146b6163752fb24f4 Sep 07 04:09:57.589694 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:57.589661 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:09:57.823772 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:57.823684 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" event={"ID":"44cb2d92-20ba-48ee-9f43-b91573487691","Type":"ContainerStarted","Data":"30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4"} Sep 07 04:09:57.823772 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:57.823741 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" event={"ID":"44cb2d92-20ba-48ee-9f43-b91573487691","Type":"ContainerStarted","Data":"d6f9a4546956942d5a39472014e0f822af9ac67dfc62d9c146b6163752fb24f4"} Sep 07 04:09:57.823994 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:57.823785 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:09:57.841455 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:09:57.841407 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" podStartSLOduration=2.448913853 podStartE2EDuration="2.84139265s" totalImagesPullingTime="392.478797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:09:57.839202835 +0000 UTC m=+1316.414069125" watchObservedRunningTime="2026-09-07 04:09:57.84139265 +0000 UTC m=+1316.416258939" Sep 07 04:10:03.834820 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:03.834787 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:10:10.394535 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:10.394500 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7356-76bc53"] Sep 07 04:10:10.399554 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:10.399530 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-76bc53" Sep 07 04:10:10.401778 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:10.401714 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7356-76bc53"] Sep 07 04:10:10.401915 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:10.401895 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhk6n\"" Sep 07 04:10:10.461816 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:10.461771 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46fwn\" (UniqueName: \"kubernetes.io/projected/74691434-b974-4d59-bc04-24dbf542cf07-kube-api-access-46fwn\") pod \"mt-curl-7356-76bc53\" (UID: \"74691434-b974-4d59-bc04-24dbf542cf07\") " pod="opendatahub/mt-curl-7356-76bc53" Sep 07 04:10:10.562811 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:10.562774 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46fwn\" (UniqueName: \"kubernetes.io/projected/74691434-b974-4d59-bc04-24dbf542cf07-kube-api-access-46fwn\") pod \"mt-curl-7356-76bc53\" (UID: \"74691434-b974-4d59-bc04-24dbf542cf07\") " pod="opendatahub/mt-curl-7356-76bc53" Sep 07 04:10:10.573118 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:10.573087 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46fwn\" (UniqueName: \"kubernetes.io/projected/74691434-b974-4d59-bc04-24dbf542cf07-kube-api-access-46fwn\") pod \"mt-curl-7356-76bc53\" (UID: \"74691434-b974-4d59-bc04-24dbf542cf07\") " pod="opendatahub/mt-curl-7356-76bc53" Sep 07 04:10:10.710346 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:10.710312 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-76bc53" Sep 07 04:10:11.000552 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:11.000480 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7356-76bc53"] Sep 07 04:10:11.003509 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:10:11.003471 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74691434_b974_4d59_bc04_24dbf542cf07.slice/crio-c6f263f4e696843cb4fe2d09668854daba9318e9d6188e7117790e49701f8eca WatchSource:0}: Error finding container c6f263f4e696843cb4fe2d09668854daba9318e9d6188e7117790e49701f8eca: Status 404 returned error can't find the container with id c6f263f4e696843cb4fe2d09668854daba9318e9d6188e7117790e49701f8eca Sep 07 04:10:11.902250 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:11.902163 2555 generic.go:356] "Generic (PLEG): container finished" podID="74691434-b974-4d59-bc04-24dbf542cf07" containerID="fac7b5ed2f5f6fa530548edb0029abc3d91fa257c2c73dc8ae842a678b48ba19" exitCode=0 Sep 07 04:10:11.902250 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:11.902220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7356-76bc53" event={"ID":"74691434-b974-4d59-bc04-24dbf542cf07","Type":"ContainerDied","Data":"fac7b5ed2f5f6fa530548edb0029abc3d91fa257c2c73dc8ae842a678b48ba19"} Sep 07 04:10:11.902665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:11.902258 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7356-76bc53" event={"ID":"74691434-b974-4d59-bc04-24dbf542cf07","Type":"ContainerStarted","Data":"c6f263f4e696843cb4fe2d09668854daba9318e9d6188e7117790e49701f8eca"} Sep 07 04:10:13.086364 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.086336 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-76bc53" Sep 07 04:10:13.104410 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.104385 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7356-76bc53_74691434-b974-4d59-bc04-24dbf542cf07/mt-curl-7356-76bc53/0.log" Sep 07 04:10:13.126259 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.126235 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7356-76bc53"] Sep 07 04:10:13.128358 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.128334 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7356-76bc53"] Sep 07 04:10:13.187736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.187651 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46fwn\" (UniqueName: \"kubernetes.io/projected/74691434-b974-4d59-bc04-24dbf542cf07-kube-api-access-46fwn\") pod \"74691434-b974-4d59-bc04-24dbf542cf07\" (UID: \"74691434-b974-4d59-bc04-24dbf542cf07\") " Sep 07 04:10:13.189698 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.189677 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74691434-b974-4d59-bc04-24dbf542cf07-kube-api-access-46fwn" (OuterVolumeSpecName: "kube-api-access-46fwn") pod "74691434-b974-4d59-bc04-24dbf542cf07" (UID: "74691434-b974-4d59-bc04-24dbf542cf07"). InnerVolumeSpecName "kube-api-access-46fwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:10:13.253146 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.253102 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7356-3cc744"] Sep 07 04:10:13.253947 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.253908 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74691434-b974-4d59-bc04-24dbf542cf07" containerName="mt-curl-7356-76bc53" Sep 07 04:10:13.254076 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.253949 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="74691434-b974-4d59-bc04-24dbf542cf07" containerName="mt-curl-7356-76bc53" Sep 07 04:10:13.254151 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.254101 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="74691434-b974-4d59-bc04-24dbf542cf07" containerName="mt-curl-7356-76bc53" Sep 07 04:10:13.257979 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.257953 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-3cc744" Sep 07 04:10:13.261504 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.261478 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7356-3cc744"] Sep 07 04:10:13.289533 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.289503 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwqwl\" (UniqueName: \"kubernetes.io/projected/688bb744-c31a-46eb-baa4-6415d1b98254-kube-api-access-mwqwl\") pod \"mt-curl-7356-3cc744\" (UID: \"688bb744-c31a-46eb-baa4-6415d1b98254\") " pod="opendatahub/mt-curl-7356-3cc744" Sep 07 04:10:13.289647 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.289629 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46fwn\" (UniqueName: \"kubernetes.io/projected/74691434-b974-4d59-bc04-24dbf542cf07-kube-api-access-46fwn\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:10:13.390165 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.390132 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwqwl\" (UniqueName: \"kubernetes.io/projected/688bb744-c31a-46eb-baa4-6415d1b98254-kube-api-access-mwqwl\") pod \"mt-curl-7356-3cc744\" (UID: \"688bb744-c31a-46eb-baa4-6415d1b98254\") " pod="opendatahub/mt-curl-7356-3cc744" Sep 07 04:10:13.398027 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.398000 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwqwl\" (UniqueName: \"kubernetes.io/projected/688bb744-c31a-46eb-baa4-6415d1b98254-kube-api-access-mwqwl\") pod \"mt-curl-7356-3cc744\" (UID: \"688bb744-c31a-46eb-baa4-6415d1b98254\") " pod="opendatahub/mt-curl-7356-3cc744" Sep 07 04:10:13.569117 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.569080 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-3cc744" Sep 07 04:10:13.838753 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.838659 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7356-3cc744"] Sep 07 04:10:13.865706 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:10:13.865675 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod688bb744_c31a_46eb_baa4_6415d1b98254.slice/crio-688b177aeb7653c58719e621a56d038fee0283c944cd71eefb3eb9af4c9ac6cd WatchSource:0}: Error finding container 688b177aeb7653c58719e621a56d038fee0283c944cd71eefb3eb9af4c9ac6cd: Status 404 returned error can't find the container with id 688b177aeb7653c58719e621a56d038fee0283c944cd71eefb3eb9af4c9ac6cd Sep 07 04:10:13.867256 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.867239 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 04:10:13.917933 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.917910 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7356-3cc744" event={"ID":"688bb744-c31a-46eb-baa4-6415d1b98254","Type":"ContainerStarted","Data":"688b177aeb7653c58719e621a56d038fee0283c944cd71eefb3eb9af4c9ac6cd"} Sep 07 04:10:13.919084 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.919045 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6f263f4e696843cb4fe2d09668854daba9318e9d6188e7117790e49701f8eca" Sep 07 04:10:13.919178 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.919087 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-76bc53" Sep 07 04:10:13.929027 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:10:13.929002 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7356-76bc53\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="74691434-b974-4d59-bc04-24dbf542cf07" pod="opendatahub/mt-curl-7356-76bc53" Sep 07 04:10:13.951377 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:13.951343 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74691434-b974-4d59-bc04-24dbf542cf07" path="/var/lib/kubelet/pods/74691434-b974-4d59-bc04-24dbf542cf07/volumes" Sep 07 04:10:14.927797 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:14.927762 2555 generic.go:356] "Generic (PLEG): container finished" podID="688bb744-c31a-46eb-baa4-6415d1b98254" containerID="33c465f901ab27297a98a9adde8286e313544bbff466cd465b80e75810ad81e7" exitCode=0 Sep 07 04:10:14.928166 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:14.927843 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7356-3cc744" event={"ID":"688bb744-c31a-46eb-baa4-6415d1b98254","Type":"ContainerDied","Data":"33c465f901ab27297a98a9adde8286e313544bbff466cd465b80e75810ad81e7"} Sep 07 04:10:16.079500 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:16.079471 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-3cc744" Sep 07 04:10:16.098821 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:16.098794 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7356-3cc744_688bb744-c31a-46eb-baa4-6415d1b98254/mt-curl-7356-3cc744/0.log" Sep 07 04:10:16.117137 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:16.117103 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwqwl\" (UniqueName: \"kubernetes.io/projected/688bb744-c31a-46eb-baa4-6415d1b98254-kube-api-access-mwqwl\") pod \"688bb744-c31a-46eb-baa4-6415d1b98254\" (UID: \"688bb744-c31a-46eb-baa4-6415d1b98254\") " Sep 07 04:10:16.119787 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:16.119754 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688bb744-c31a-46eb-baa4-6415d1b98254-kube-api-access-mwqwl" (OuterVolumeSpecName: "kube-api-access-mwqwl") pod "688bb744-c31a-46eb-baa4-6415d1b98254" (UID: "688bb744-c31a-46eb-baa4-6415d1b98254"). InnerVolumeSpecName "kube-api-access-mwqwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:10:16.121317 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:16.121288 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7356-3cc744"] Sep 07 04:10:16.124899 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:16.124873 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7356-3cc744"] Sep 07 04:10:16.218671 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:16.218638 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwqwl\" (UniqueName: \"kubernetes.io/projected/688bb744-c31a-46eb-baa4-6415d1b98254-kube-api-access-mwqwl\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:10:16.941524 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:16.941494 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7356-3cc744" Sep 07 04:10:16.941524 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:16.941518 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="688b177aeb7653c58719e621a56d038fee0283c944cd71eefb3eb9af4c9ac6cd" Sep 07 04:10:16.951205 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:10:16.951176 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7356-3cc744\" is forbidden: User \"system:node:ip-10-0-131-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-215.ec2.internal' and this object" podUID="688bb744-c31a-46eb-baa4-6415d1b98254" pod="opendatahub/mt-curl-7356-3cc744" Sep 07 04:10:17.937763 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:17.937734 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="688bb744-c31a-46eb-baa4-6415d1b98254" path="/var/lib/kubelet/pods/688bb744-c31a-46eb-baa4-6415d1b98254/volumes" Sep 07 04:10:17.948221 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:17.948197 2555 generic.go:356] "Generic (PLEG): container finished" podID="b8a28179-10cf-44db-94f2-c4b06d591074" containerID="36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c" exitCode=0 Sep 07 04:10:17.948341 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:17.948227 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" event={"ID":"b8a28179-10cf-44db-94f2-c4b06d591074","Type":"ContainerDied","Data":"36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c"} Sep 07 04:10:17.948561 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:17.948548 2555 scope.go:117] "RemoveContainer" containerID="36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c" Sep 07 04:10:17.949600 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:17.949573 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:10:17.949654 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:17.949640 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:10:17.949693 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:17.949681 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:10:17.999527 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:17.999500 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:10:18.960990 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:18.960948 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" event={"ID":"b8a28179-10cf-44db-94f2-c4b06d591074","Type":"ContainerStarted","Data":"5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7"} Sep 07 04:10:18.967342 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:18.967316 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:10:19.968267 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:19.968233 2555 patch_prober.go:28] interesting pod/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 07 04:10:19.968771 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:19.968294 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 07 04:10:20.460133 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.450401 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb"] Sep 07 04:10:20.460133 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.451079 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688bb744-c31a-46eb-baa4-6415d1b98254" containerName="mt-curl-7356-3cc744" Sep 07 04:10:20.460133 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.451097 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="688bb744-c31a-46eb-baa4-6415d1b98254" containerName="mt-curl-7356-3cc744" Sep 07 04:10:20.460133 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.451219 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="688bb744-c31a-46eb-baa4-6415d1b98254" containerName="mt-curl-7356-3cc744" Sep 07 04:10:20.467442 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.467421 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.471154 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.471137 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cf1797-openshift-default-dockercfg-zs7cm\"" Sep 07 04:10:20.480459 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.480436 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb"] Sep 07 04:10:20.567108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.566580 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-socket\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.567108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.566675 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-data\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.567108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.566701 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-istio-token\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.567108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.566737 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-certs\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.567108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.566765 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-credential-socket\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.567108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.566798 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l72t\" (UniqueName: \"kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-kube-api-access-8l72t\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.567108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.566864 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b858cfe1-908c-436b-b568-b0a32b44f661-istiod-ca-cert\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.567108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.566951 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b858cfe1-908c-436b-b568-b0a32b44f661-istio-podinfo\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.567108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.566976 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-envoy\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668133 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668099 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b858cfe1-908c-436b-b568-b0a32b44f661-istio-podinfo\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668133 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668137 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-envoy\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668400 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668160 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-socket\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668400 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668232 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-data\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668400 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668255 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-istio-token\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668400 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668292 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-certs\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668400 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668316 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-credential-socket\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668400 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668347 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l72t\" (UniqueName: \"kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-kube-api-access-8l72t\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668727 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668414 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b858cfe1-908c-436b-b568-b0a32b44f661-istiod-ca-cert\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668727 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668642 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-data\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668727 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668661 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-socket\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668900 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668876 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-credential-socket\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.668963 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.668910 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-certs\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.669348 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.669326 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b858cfe1-908c-436b-b568-b0a32b44f661-istiod-ca-cert\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.670534 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.670502 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-envoy\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.670737 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.670718 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b858cfe1-908c-436b-b568-b0a32b44f661-istio-podinfo\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.677963 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.677940 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-istio-token\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.678732 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.678709 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l72t\" (UniqueName: \"kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-kube-api-access-8l72t\") pod \"e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.781652 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.781569 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:20.967630 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.967594 2555 patch_prober.go:28] interesting pod/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 07 04:10:20.967791 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:20.967648 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 07 04:10:21.076481 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:21.076455 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb"] Sep 07 04:10:21.078166 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:10:21.078130 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb858cfe1_908c_436b_b568_b0a32b44f661.slice/crio-b58ffe056478515d3c4bd5fa0b0b2e8d9f956ba1800b4ed3b9767250b91faa5e WatchSource:0}: Error finding container b58ffe056478515d3c4bd5fa0b0b2e8d9f956ba1800b4ed3b9767250b91faa5e: Status 404 returned error can't find the container with id b58ffe056478515d3c4bd5fa0b0b2e8d9f956ba1800b4ed3b9767250b91faa5e Sep 07 04:10:21.080775 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:21.080742 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:10:21.080857 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:21.080808 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:10:21.080857 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:21.080834 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:10:21.967727 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:21.967684 2555 patch_prober.go:28] interesting pod/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 07 04:10:21.967970 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:21.967746 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 07 04:10:21.979839 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:21.979803 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" event={"ID":"b858cfe1-908c-436b-b568-b0a32b44f661","Type":"ContainerStarted","Data":"e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1"} Sep 07 04:10:21.980003 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:21.979859 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" event={"ID":"b858cfe1-908c-436b-b568-b0a32b44f661","Type":"ContainerStarted","Data":"b58ffe056478515d3c4bd5fa0b0b2e8d9f956ba1800b4ed3b9767250b91faa5e"} Sep 07 04:10:21.998478 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:21.998426 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" podStartSLOduration=1.9984112550000002 podStartE2EDuration="1.998411255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:10:21.997289399 +0000 UTC m=+1340.572155710" watchObservedRunningTime="2026-09-07 04:10:21.998411255 +0000 UTC m=+1340.573277613" Sep 07 04:10:22.782074 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:22.782019 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:22.787218 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:22.787194 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:22.967933 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:22.967892 2555 patch_prober.go:28] interesting pod/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 07 04:10:22.968127 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:22.967950 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 07 04:10:22.987048 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:22.987019 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:22.988115 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:22.988092 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:23.968503 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:23.968467 2555 patch_prober.go:28] interesting pod/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 07 04:10:23.969012 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:23.968525 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 07 04:10:24.806977 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:24.806939 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:10:24.972162 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:24.972133 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:10:24.998051 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:24.998023 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:10:35.091978 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:35.091939 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:10:45.110174 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:45.110147 2555 generic.go:356] "Generic (PLEG): container finished" podID="b858cfe1-908c-436b-b568-b0a32b44f661" containerID="e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1" exitCode=0 Sep 07 04:10:45.110547 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:45.110220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" event={"ID":"b858cfe1-908c-436b-b568-b0a32b44f661","Type":"ContainerDied","Data":"e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1"} Sep 07 04:10:45.110641 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:45.110626 2555 scope.go:117] "RemoveContainer" containerID="e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1" Sep 07 04:10:45.111762 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:45.111737 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:10:45.111820 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:45.111801 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:10:45.111859 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:45.111834 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:10:46.117317 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:46.117280 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" event={"ID":"b858cfe1-908c-436b-b568-b0a32b44f661","Type":"ContainerStarted","Data":"7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b"} Sep 07 04:10:46.782451 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:46.782405 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:46.782636 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:46.782608 2555 patch_prober.go:28] interesting pod/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 07 04:10:46.782682 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:46.782654 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 07 04:10:47.782876 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:47.782838 2555 patch_prober.go:28] interesting pod/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 07 04:10:47.783286 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:47.782898 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 07 04:10:48.782936 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:48.782896 2555 patch_prober.go:28] interesting pod/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 07 04:10:48.783369 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:48.782970 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 07 04:10:49.782643 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:49.782606 2555 patch_prober.go:28] interesting pod/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 07 04:10:49.782833 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:49.782673 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 07 04:10:50.782808 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:50.782774 2555 patch_prober.go:28] interesting pod/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 07 04:10:50.783225 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:50.782833 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 07 04:10:51.786921 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:51.786881 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:51.787406 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:51.787199 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:51.787822 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:51.787807 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:10:55.696396 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:10:55.696361 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:11:02.152285 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:02.152251 2555 scope.go:117] "RemoveContainer" containerID="d3561225f63cc836809abfd472d04f6b2f2f168439a25a720e8d26587493e0eb" Sep 07 04:11:07.174527 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.174490 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v"] Sep 07 04:11:07.175006 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.174746 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" podUID="44cb2d92-20ba-48ee-9f43-b91573487691" containerName="maas-api" containerID="cri-o://30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4" gracePeriod=30 Sep 07 04:11:07.430381 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.430317 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:11:07.547305 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.547269 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cb2d92-20ba-48ee-9f43-b91573487691-openshift-service-ca\") pod \"44cb2d92-20ba-48ee-9f43-b91573487691\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " Sep 07 04:11:07.547496 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.547371 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9gxk\" (UniqueName: \"kubernetes.io/projected/44cb2d92-20ba-48ee-9f43-b91573487691-kube-api-access-p9gxk\") pod \"44cb2d92-20ba-48ee-9f43-b91573487691\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " Sep 07 04:11:07.547496 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.547400 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-metrics-certs\") pod \"44cb2d92-20ba-48ee-9f43-b91573487691\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " Sep 07 04:11:07.547496 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.547473 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-maas-api-tls\") pod \"44cb2d92-20ba-48ee-9f43-b91573487691\" (UID: \"44cb2d92-20ba-48ee-9f43-b91573487691\") " Sep 07 04:11:07.547718 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.547692 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cb2d92-20ba-48ee-9f43-b91573487691-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44cb2d92-20ba-48ee-9f43-b91573487691" (UID: "44cb2d92-20ba-48ee-9f43-b91573487691"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:11:07.549461 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.549424 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "44cb2d92-20ba-48ee-9f43-b91573487691" (UID: "44cb2d92-20ba-48ee-9f43-b91573487691"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:11:07.549575 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.549515 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cb2d92-20ba-48ee-9f43-b91573487691-kube-api-access-p9gxk" (OuterVolumeSpecName: "kube-api-access-p9gxk") pod "44cb2d92-20ba-48ee-9f43-b91573487691" (UID: "44cb2d92-20ba-48ee-9f43-b91573487691"). InnerVolumeSpecName "kube-api-access-p9gxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:11:07.549575 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.549513 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44cb2d92-20ba-48ee-9f43-b91573487691" (UID: "44cb2d92-20ba-48ee-9f43-b91573487691"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:11:07.648967 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.648935 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-maas-api-tls\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:07.648967 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.648960 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cb2d92-20ba-48ee-9f43-b91573487691-openshift-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:07.648967 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.648969 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9gxk\" (UniqueName: \"kubernetes.io/projected/44cb2d92-20ba-48ee-9f43-b91573487691-kube-api-access-p9gxk\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:07.649210 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:07.648977 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44cb2d92-20ba-48ee-9f43-b91573487691-metrics-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:08.240386 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:08.240345 2555 generic.go:356] "Generic (PLEG): container finished" podID="44cb2d92-20ba-48ee-9f43-b91573487691" containerID="30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4" exitCode=0 Sep 07 04:11:08.240824 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:08.240390 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" Sep 07 04:11:08.240824 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:08.240419 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" event={"ID":"44cb2d92-20ba-48ee-9f43-b91573487691","Type":"ContainerDied","Data":"30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4"} Sep 07 04:11:08.240824 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:08.240468 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v" event={"ID":"44cb2d92-20ba-48ee-9f43-b91573487691","Type":"ContainerDied","Data":"d6f9a4546956942d5a39472014e0f822af9ac67dfc62d9c146b6163752fb24f4"} Sep 07 04:11:08.240824 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:08.240490 2555 scope.go:117] "RemoveContainer" containerID="30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4" Sep 07 04:11:08.251176 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:08.251082 2555 scope.go:117] "RemoveContainer" containerID="30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4" Sep 07 04:11:08.251534 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:11:08.251463 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4\": container with ID starting with 30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4 not found: ID does not exist" containerID="30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4" Sep 07 04:11:08.251534 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:08.251497 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4"} err="failed to get container status \"30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4\": rpc error: code = NotFound desc = could not find container \"30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4\": container with ID starting with 30349df7355dee0d542f3b1079380d9f79993e2d10e57ca9281ef0444a3092b4 not found: ID does not exist" Sep 07 04:11:08.258016 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:08.257993 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v"] Sep 07 04:11:08.261298 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:08.261279 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-467e44-7f5d95487b-lx45v"] Sep 07 04:11:09.937501 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:09.937468 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44cb2d92-20ba-48ee-9f43-b91573487691" path="/var/lib/kubelet/pods/44cb2d92-20ba-48ee-9f43-b91573487691/volumes" Sep 07 04:11:13.694872 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:13.694841 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht"] Sep 07 04:11:13.695365 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:13.695095 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" containerID="cri-o://5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7" gracePeriod=30 Sep 07 04:11:15.089313 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:15.089271 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:11:18.951135 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:18.951114 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:11:19.063285 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063204 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a28179-10cf-44db-94f2-c4b06d591074-istio-podinfo\") pod \"b8a28179-10cf-44db-94f2-c4b06d591074\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " Sep 07 04:11:19.063285 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063249 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-credential-socket\") pod \"b8a28179-10cf-44db-94f2-c4b06d591074\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " Sep 07 04:11:19.063285 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063277 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-certs\") pod \"b8a28179-10cf-44db-94f2-c4b06d591074\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " Sep 07 04:11:19.063555 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063298 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-envoy\") pod \"b8a28179-10cf-44db-94f2-c4b06d591074\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " Sep 07 04:11:19.063555 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063320 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-data\") pod \"b8a28179-10cf-44db-94f2-c4b06d591074\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " Sep 07 04:11:19.063555 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063346 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-istio-token\") pod \"b8a28179-10cf-44db-94f2-c4b06d591074\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " Sep 07 04:11:19.063555 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063383 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkgp5\" (UniqueName: \"kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-kube-api-access-tkgp5\") pod \"b8a28179-10cf-44db-94f2-c4b06d591074\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " Sep 07 04:11:19.063555 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063439 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a28179-10cf-44db-94f2-c4b06d591074-istiod-ca-cert\") pod \"b8a28179-10cf-44db-94f2-c4b06d591074\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " Sep 07 04:11:19.063555 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063480 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-socket\") pod \"b8a28179-10cf-44db-94f2-c4b06d591074\" (UID: \"b8a28179-10cf-44db-94f2-c4b06d591074\") " Sep 07 04:11:19.063555 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063504 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b8a28179-10cf-44db-94f2-c4b06d591074" (UID: "b8a28179-10cf-44db-94f2-c4b06d591074"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:11:19.063906 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063715 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-data" (OuterVolumeSpecName: "istio-data") pod "b8a28179-10cf-44db-94f2-c4b06d591074" (UID: "b8a28179-10cf-44db-94f2-c4b06d591074"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:11:19.063906 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063868 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b8a28179-10cf-44db-94f2-c4b06d591074" (UID: "b8a28179-10cf-44db-94f2-c4b06d591074"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:11:19.063906 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063893 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a28179-10cf-44db-94f2-c4b06d591074-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b8a28179-10cf-44db-94f2-c4b06d591074" (UID: "b8a28179-10cf-44db-94f2-c4b06d591074"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:11:19.064088 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063977 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:19.064088 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.063996 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:19.064088 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.064011 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a28179-10cf-44db-94f2-c4b06d591074-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:19.064088 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.064024 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:19.064310 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.064260 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b8a28179-10cf-44db-94f2-c4b06d591074" (UID: "b8a28179-10cf-44db-94f2-c4b06d591074"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:11:19.065663 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.065629 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8a28179-10cf-44db-94f2-c4b06d591074-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b8a28179-10cf-44db-94f2-c4b06d591074" (UID: "b8a28179-10cf-44db-94f2-c4b06d591074"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:11:19.065778 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.065693 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b8a28179-10cf-44db-94f2-c4b06d591074" (UID: "b8a28179-10cf-44db-94f2-c4b06d591074"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:11:19.065829 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.065809 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-kube-api-access-tkgp5" (OuterVolumeSpecName: "kube-api-access-tkgp5") pod "b8a28179-10cf-44db-94f2-c4b06d591074" (UID: "b8a28179-10cf-44db-94f2-c4b06d591074"). InnerVolumeSpecName "kube-api-access-tkgp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:11:19.065912 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.065898 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-istio-token" (OuterVolumeSpecName: "istio-token") pod "b8a28179-10cf-44db-94f2-c4b06d591074" (UID: "b8a28179-10cf-44db-94f2-c4b06d591074"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:11:19.164914 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.164889 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:19.164914 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.164911 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkgp5\" (UniqueName: \"kubernetes.io/projected/b8a28179-10cf-44db-94f2-c4b06d591074-kube-api-access-tkgp5\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:19.164914 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.164921 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a28179-10cf-44db-94f2-c4b06d591074-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:19.164914 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.164930 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:19.165263 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.164938 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a28179-10cf-44db-94f2-c4b06d591074-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:19.302173 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.302139 2555 generic.go:356] "Generic (PLEG): container finished" podID="b8a28179-10cf-44db-94f2-c4b06d591074" containerID="5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7" exitCode=0 Sep 07 04:11:19.302328 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.302183 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" event={"ID":"b8a28179-10cf-44db-94f2-c4b06d591074","Type":"ContainerDied","Data":"5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7"} Sep 07 04:11:19.302328 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.302239 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" event={"ID":"b8a28179-10cf-44db-94f2-c4b06d591074","Type":"ContainerDied","Data":"e57b42f1c5981ced6e9c0810d862e9fc20d82714f7199d44a68fb9ce469ba65a"} Sep 07 04:11:19.302328 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.302262 2555 scope.go:117] "RemoveContainer" containerID="5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7" Sep 07 04:11:19.302328 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.302192 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht" Sep 07 04:11:19.312284 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.312264 2555 scope.go:117] "RemoveContainer" containerID="36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c" Sep 07 04:11:19.320166 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.320150 2555 scope.go:117] "RemoveContainer" containerID="5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7" Sep 07 04:11:19.320402 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:11:19.320382 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7\": container with ID starting with 5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7 not found: ID does not exist" containerID="5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7" Sep 07 04:11:19.320455 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.320409 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7"} err="failed to get container status \"5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7\": rpc error: code = NotFound desc = could not find container \"5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7\": container with ID starting with 5f34a934d72b0f5b00b21a5da1cc91c168a8ca567df814e5ebfa9b501a5c87d7 not found: ID does not exist" Sep 07 04:11:19.320455 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.320429 2555 scope.go:117] "RemoveContainer" containerID="36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c" Sep 07 04:11:19.320677 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:11:19.320660 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c\": container with ID starting with 36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c not found: ID does not exist" containerID="36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c" Sep 07 04:11:19.320717 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.320682 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c"} err="failed to get container status \"36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c\": rpc error: code = NotFound desc = could not find container \"36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c\": container with ID starting with 36618841e9caa26d1974182f147cdf9f0f7237a7c3a7a75a2463bc4772982b0c not found: ID does not exist" Sep 07 04:11:19.324258 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.324237 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht"] Sep 07 04:11:19.327722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.327702 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-467e44-openshift-default-7d7bd4988d-kxbht"] Sep 07 04:11:19.938758 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:19.938725 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" path="/var/lib/kubelet/pods/b8a28179-10cf-44db-94f2-c4b06d591074/volumes" Sep 07 04:11:27.393407 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:27.393373 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:11:35.982874 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:35.982808 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb"] Sep 07 04:11:35.983373 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:35.983208 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" containerID="cri-o://7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b" gracePeriod=30 Sep 07 04:11:36.789600 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:36.789563 2555 patch_prober.go:28] interesting pod/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 04:11:36.789784 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:36.789629 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 04:11:41.250320 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.250260 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:11:41.378079 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378015 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b858cfe1-908c-436b-b568-b0a32b44f661-istio-podinfo\") pod \"b858cfe1-908c-436b-b568-b0a32b44f661\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " Sep 07 04:11:41.378251 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378116 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-certs\") pod \"b858cfe1-908c-436b-b568-b0a32b44f661\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " Sep 07 04:11:41.378251 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378143 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l72t\" (UniqueName: \"kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-kube-api-access-8l72t\") pod \"b858cfe1-908c-436b-b568-b0a32b44f661\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " Sep 07 04:11:41.378251 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378176 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-credential-socket\") pod \"b858cfe1-908c-436b-b568-b0a32b44f661\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " Sep 07 04:11:41.378251 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378237 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b858cfe1-908c-436b-b568-b0a32b44f661-istiod-ca-cert\") pod \"b858cfe1-908c-436b-b568-b0a32b44f661\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " Sep 07 04:11:41.378529 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378263 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-data\") pod \"b858cfe1-908c-436b-b568-b0a32b44f661\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " Sep 07 04:11:41.378529 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378335 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-socket\") pod \"b858cfe1-908c-436b-b568-b0a32b44f661\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " Sep 07 04:11:41.378529 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378377 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-envoy\") pod \"b858cfe1-908c-436b-b568-b0a32b44f661\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " Sep 07 04:11:41.378529 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378437 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-istio-token\") pod \"b858cfe1-908c-436b-b568-b0a32b44f661\" (UID: \"b858cfe1-908c-436b-b568-b0a32b44f661\") " Sep 07 04:11:41.378529 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378458 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b858cfe1-908c-436b-b568-b0a32b44f661" (UID: "b858cfe1-908c-436b-b568-b0a32b44f661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:11:41.378885 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378553 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b858cfe1-908c-436b-b568-b0a32b44f661" (UID: "b858cfe1-908c-436b-b568-b0a32b44f661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:11:41.378885 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378594 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b858cfe1-908c-436b-b568-b0a32b44f661" (UID: "b858cfe1-908c-436b-b568-b0a32b44f661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:11:41.378885 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378672 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b858cfe1-908c-436b-b568-b0a32b44f661-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b858cfe1-908c-436b-b568-b0a32b44f661" (UID: "b858cfe1-908c-436b-b568-b0a32b44f661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:11:41.378885 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378846 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-data" (OuterVolumeSpecName: "istio-data") pod "b858cfe1-908c-436b-b568-b0a32b44f661" (UID: "b858cfe1-908c-436b-b568-b0a32b44f661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:11:41.379082 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378912 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:41.379082 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378933 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:41.379082 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378947 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b858cfe1-908c-436b-b568-b0a32b44f661-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:41.379082 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.378960 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:41.380424 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.380400 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b858cfe1-908c-436b-b568-b0a32b44f661-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b858cfe1-908c-436b-b568-b0a32b44f661" (UID: "b858cfe1-908c-436b-b568-b0a32b44f661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:11:41.380424 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.380404 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-kube-api-access-8l72t" (OuterVolumeSpecName: "kube-api-access-8l72t") pod "b858cfe1-908c-436b-b568-b0a32b44f661" (UID: "b858cfe1-908c-436b-b568-b0a32b44f661"). InnerVolumeSpecName "kube-api-access-8l72t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:11:41.380846 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.380831 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b858cfe1-908c-436b-b568-b0a32b44f661" (UID: "b858cfe1-908c-436b-b568-b0a32b44f661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:11:41.380892 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.380862 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-istio-token" (OuterVolumeSpecName: "istio-token") pod "b858cfe1-908c-436b-b568-b0a32b44f661" (UID: "b858cfe1-908c-436b-b568-b0a32b44f661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:11:41.428635 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.428602 2555 generic.go:356] "Generic (PLEG): container finished" podID="b858cfe1-908c-436b-b568-b0a32b44f661" containerID="7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b" exitCode=0 Sep 07 04:11:41.428796 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.428662 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" Sep 07 04:11:41.428796 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.428689 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" event={"ID":"b858cfe1-908c-436b-b568-b0a32b44f661","Type":"ContainerDied","Data":"7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b"} Sep 07 04:11:41.428796 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.428777 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb" event={"ID":"b858cfe1-908c-436b-b568-b0a32b44f661","Type":"ContainerDied","Data":"b58ffe056478515d3c4bd5fa0b0b2e8d9f956ba1800b4ed3b9767250b91faa5e"} Sep 07 04:11:41.428945 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.428800 2555 scope.go:117] "RemoveContainer" containerID="7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b" Sep 07 04:11:41.438882 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.438859 2555 scope.go:117] "RemoveContainer" containerID="e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1" Sep 07 04:11:41.447034 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.447017 2555 scope.go:117] "RemoveContainer" containerID="7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b" Sep 07 04:11:41.447308 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:11:41.447289 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b\": container with ID starting with 7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b not found: ID does not exist" containerID="7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b" Sep 07 04:11:41.447364 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.447315 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b"} err="failed to get container status \"7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b\": rpc error: code = NotFound desc = could not find container \"7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b\": container with ID starting with 7193b2df2531280fbff2159c0a2125de4ffeaa5094d28c19f92044b6de8a0e3b not found: ID does not exist" Sep 07 04:11:41.447364 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.447332 2555 scope.go:117] "RemoveContainer" containerID="e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1" Sep 07 04:11:41.447569 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:11:41.447552 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1\": container with ID starting with e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1 not found: ID does not exist" containerID="e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1" Sep 07 04:11:41.447622 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.447578 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1"} err="failed to get container status \"e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1\": rpc error: code = NotFound desc = could not find container \"e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1\": container with ID starting with e3d02c67922aa0b5593a4fbf60cfcaff0a2cf83800b68ff596cbed663f6c0fd1 not found: ID does not exist" Sep 07 04:11:41.451550 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.451531 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb"] Sep 07 04:11:41.453314 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.453297 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cf1797-openshift-default-7ddf96f9cb-bqmkb"] Sep 07 04:11:41.479653 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.479634 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:41.479653 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.479654 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:41.479792 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.479662 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b858cfe1-908c-436b-b568-b0a32b44f661-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:41.479792 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.479670 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l72t\" (UniqueName: \"kubernetes.io/projected/b858cfe1-908c-436b-b568-b0a32b44f661-kube-api-access-8l72t\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:41.479792 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.479680 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b858cfe1-908c-436b-b568-b0a32b44f661-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:11:41.936265 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:41.936230 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" path="/var/lib/kubelet/pods/b858cfe1-908c-436b-b568-b0a32b44f661/volumes" Sep 07 04:11:42.788616 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:42.788583 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:11:57.794073 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:11:57.793983 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:12:03.885140 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:03.885106 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp"] Sep 07 04:12:03.885596 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:03.885360 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" podUID="3cd02547-b07d-4d83-ba95-6a586bc74dde" containerName="maas-api" containerID="cri-o://0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb" gracePeriod=30 Sep 07 04:12:04.130005 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.129977 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:12:04.184348 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.184273 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxwct\" (UniqueName: \"kubernetes.io/projected/3cd02547-b07d-4d83-ba95-6a586bc74dde-kube-api-access-hxwct\") pod \"3cd02547-b07d-4d83-ba95-6a586bc74dde\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " Sep 07 04:12:04.184348 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.184308 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-maas-api-tls\") pod \"3cd02547-b07d-4d83-ba95-6a586bc74dde\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " Sep 07 04:12:04.184348 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.184329 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd02547-b07d-4d83-ba95-6a586bc74dde-openshift-service-ca\") pod \"3cd02547-b07d-4d83-ba95-6a586bc74dde\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " Sep 07 04:12:04.184604 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.184442 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-metrics-certs\") pod \"3cd02547-b07d-4d83-ba95-6a586bc74dde\" (UID: \"3cd02547-b07d-4d83-ba95-6a586bc74dde\") " Sep 07 04:12:04.184713 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.184693 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd02547-b07d-4d83-ba95-6a586bc74dde-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cd02547-b07d-4d83-ba95-6a586bc74dde" (UID: "3cd02547-b07d-4d83-ba95-6a586bc74dde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:12:04.184857 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.184840 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd02547-b07d-4d83-ba95-6a586bc74dde-openshift-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:12:04.186674 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.186647 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3cd02547-b07d-4d83-ba95-6a586bc74dde" (UID: "3cd02547-b07d-4d83-ba95-6a586bc74dde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:12:04.186924 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.186905 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd02547-b07d-4d83-ba95-6a586bc74dde-kube-api-access-hxwct" (OuterVolumeSpecName: "kube-api-access-hxwct") pod "3cd02547-b07d-4d83-ba95-6a586bc74dde" (UID: "3cd02547-b07d-4d83-ba95-6a586bc74dde"). InnerVolumeSpecName "kube-api-access-hxwct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:12:04.187002 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.186935 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cd02547-b07d-4d83-ba95-6a586bc74dde" (UID: "3cd02547-b07d-4d83-ba95-6a586bc74dde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:12:04.285819 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.285787 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-metrics-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:12:04.285819 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.285812 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxwct\" (UniqueName: \"kubernetes.io/projected/3cd02547-b07d-4d83-ba95-6a586bc74dde-kube-api-access-hxwct\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:12:04.285819 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.285822 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd02547-b07d-4d83-ba95-6a586bc74dde-maas-api-tls\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:12:04.560157 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.560121 2555 generic.go:356] "Generic (PLEG): container finished" podID="3cd02547-b07d-4d83-ba95-6a586bc74dde" containerID="0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb" exitCode=0 Sep 07 04:12:04.560359 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.560169 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" Sep 07 04:12:04.560359 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.560275 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" event={"ID":"3cd02547-b07d-4d83-ba95-6a586bc74dde","Type":"ContainerDied","Data":"0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb"} Sep 07 04:12:04.560359 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.560324 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp" event={"ID":"3cd02547-b07d-4d83-ba95-6a586bc74dde","Type":"ContainerDied","Data":"19bf1fe202ca4506d407783c701e19656dce9f8298e57c75617c3a177f4d9b5c"} Sep 07 04:12:04.560359 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.560347 2555 scope.go:117] "RemoveContainer" containerID="0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb" Sep 07 04:12:04.571149 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.571101 2555 scope.go:117] "RemoveContainer" containerID="0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb" Sep 07 04:12:04.571483 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:12:04.571386 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb\": container with ID starting with 0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb not found: ID does not exist" containerID="0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb" Sep 07 04:12:04.571483 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.571423 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb"} err="failed to get container status \"0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb\": rpc error: code = NotFound desc = could not find container \"0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb\": container with ID starting with 0b66fdbe48c7f5224f3fcd399f3b26a60ac236e7690281fe6522eb909e19e4eb not found: ID does not exist" Sep 07 04:12:04.582092 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.582069 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp"] Sep 07 04:12:04.585765 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:04.585745 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a89024-5c4b47f45d-lncnp"] Sep 07 04:12:05.936384 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:05.936347 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cd02547-b07d-4d83-ba95-6a586bc74dde" path="/var/lib/kubelet/pods/3cd02547-b07d-4d83-ba95-6a586bc74dde/volumes" Sep 07 04:12:06.847427 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.847391 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4"] Sep 07 04:12:06.847900 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.847882 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" Sep 07 04:12:06.847997 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.847903 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" Sep 07 04:12:06.847997 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.847919 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" Sep 07 04:12:06.847997 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.847927 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" Sep 07 04:12:06.847997 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.847952 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd02547-b07d-4d83-ba95-6a586bc74dde" containerName="maas-api" Sep 07 04:12:06.847997 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.847961 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd02547-b07d-4d83-ba95-6a586bc74dde" containerName="maas-api" Sep 07 04:12:06.847997 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.847979 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44cb2d92-20ba-48ee-9f43-b91573487691" containerName="maas-api" Sep 07 04:12:06.847997 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.847987 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="44cb2d92-20ba-48ee-9f43-b91573487691" containerName="maas-api" Sep 07 04:12:06.848373 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.848108 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="44cb2d92-20ba-48ee-9f43-b91573487691" containerName="maas-api" Sep 07 04:12:06.848373 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.848124 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd02547-b07d-4d83-ba95-6a586bc74dde" containerName="maas-api" Sep 07 04:12:06.848373 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.848133 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" Sep 07 04:12:06.848373 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.848141 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" Sep 07 04:12:06.848373 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.848152 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" Sep 07 04:12:06.848373 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.848161 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" Sep 07 04:12:06.853108 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.853088 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:06.855459 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.855437 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f8fdfa-openshift-default-dockercfg-mrt7k\"" Sep 07 04:12:06.862872 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.862852 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4"] Sep 07 04:12:06.907781 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.907756 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-podinfo\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:06.907895 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.907796 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-credential-socket\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:06.907895 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.907813 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-envoy\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:06.907895 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.907856 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-data\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:06.907895 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.907888 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istiod-ca-cert\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:06.908044 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.907945 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4kpq\" (UniqueName: \"kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-kube-api-access-m4kpq\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:06.908044 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.907977 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-certs\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:06.908044 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.907993 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-token\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:06.908044 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:06.908025 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-socket\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.009282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009241 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-certs\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.009722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009293 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-token\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.009722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009326 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-socket\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.009722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009379 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-podinfo\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.009722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009425 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-credential-socket\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.009722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009448 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-envoy\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.009722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009542 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-data\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.009722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009626 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istiod-ca-cert\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.009722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009689 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4kpq\" (UniqueName: \"kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-kube-api-access-m4kpq\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.009722 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009699 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-certs\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.010237 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009743 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-socket\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.010237 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.009960 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-data\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.010237 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.010095 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-credential-socket\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.010406 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.010387 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istiod-ca-cert\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.011864 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.011841 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-envoy\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.012069 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.012038 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-podinfo\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.016962 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.016942 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-token\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.017149 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.017132 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4kpq\" (UniqueName: \"kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-kube-api-access-m4kpq\") pod \"e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.165464 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.165374 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:07.440363 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.440285 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4"] Sep 07 04:12:07.444008 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:12:07.443972 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9cfa32_08d4_4107_bd1b_60f1a927e471.slice/crio-8ce766caba6bdf766050beadc60f2c6da8a61e9cd13c42d7582f413879bc4576 WatchSource:0}: Error finding container 8ce766caba6bdf766050beadc60f2c6da8a61e9cd13c42d7582f413879bc4576: Status 404 returned error can't find the container with id 8ce766caba6bdf766050beadc60f2c6da8a61e9cd13c42d7582f413879bc4576 Sep 07 04:12:07.446486 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.446448 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:12:07.446579 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.446534 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:12:07.446619 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.446574 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:12:07.580173 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.580130 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" event={"ID":"2f9cfa32-08d4-4107-bd1b-60f1a927e471","Type":"ContainerStarted","Data":"82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235"} Sep 07 04:12:07.580336 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.580202 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" event={"ID":"2f9cfa32-08d4-4107-bd1b-60f1a927e471","Type":"ContainerStarted","Data":"8ce766caba6bdf766050beadc60f2c6da8a61e9cd13c42d7582f413879bc4576"} Sep 07 04:12:07.598960 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:07.598910 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" podStartSLOduration=1.598894899 podStartE2EDuration="1.598894899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:12:07.597608624 +0000 UTC m=+1446.172474914" watchObservedRunningTime="2026-09-07 04:12:07.598894899 +0000 UTC m=+1446.173761187" Sep 07 04:12:08.166243 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:08.166206 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:08.171157 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:08.171133 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:08.585233 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:08.585202 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:08.586296 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:08.586271 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:09.288362 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:09.288331 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:12:22.001853 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:22.001816 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:12:24.627263 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:24.627220 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 04:12:24.627744 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:24.627293 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 04:12:25.687815 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:25.687780 2555 generic.go:356] "Generic (PLEG): container finished" podID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerID="82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235" exitCode=0 Sep 07 04:12:25.688228 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:25.687852 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" event={"ID":"2f9cfa32-08d4-4107-bd1b-60f1a927e471","Type":"ContainerDied","Data":"82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235"} Sep 07 04:12:25.688282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:25.688261 2555 scope.go:117] "RemoveContainer" containerID="82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235" Sep 07 04:12:25.689306 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:25.689282 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:12:25.689413 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:25.689350 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:12:25.689413 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:25.689380 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:12:26.695186 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:26.695148 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" event={"ID":"2f9cfa32-08d4-4107-bd1b-60f1a927e471","Type":"ContainerStarted","Data":"3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483"} Sep 07 04:12:27.166394 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:27.166358 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:27.166607 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:27.166585 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 07 04:12:27.166659 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:27.166635 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 07 04:12:28.166593 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:28.166562 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 07 04:12:28.166984 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:28.166617 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 07 04:12:29.166353 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:29.166321 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 07 04:12:29.166527 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:29.166373 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 07 04:12:30.166101 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:30.166032 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 07 04:12:30.166507 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:30.166112 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 07 04:12:31.166460 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:31.166409 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 07 04:12:31.166835 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:31.166491 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 07 04:12:32.306028 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:32.305993 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:32.306532 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:32.306511 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:32.307641 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:32.307618 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:12:39.865461 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:39.865429 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf"] Sep 07 04:12:39.865913 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:39.865801 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" Sep 07 04:12:39.865913 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:39.865811 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a28179-10cf-44db-94f2-c4b06d591074" containerName="istio-proxy" Sep 07 04:12:39.865913 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:39.865823 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" Sep 07 04:12:39.865913 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:39.865828 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b858cfe1-908c-436b-b568-b0a32b44f661" containerName="istio-proxy" Sep 07 04:12:39.869323 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:39.869300 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:39.871651 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:39.871627 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-704b16-serving-cert\"" Sep 07 04:12:39.882111 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:39.882088 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf"] Sep 07 04:12:40.024511 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.024475 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx84r\" (UniqueName: \"kubernetes.io/projected/e2094290-5903-4a68-bbf7-78734a56887f-kube-api-access-vx84r\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.024666 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.024527 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2094290-5903-4a68-bbf7-78734a56887f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.024666 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.024606 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.024666 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.024663 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-metrics-certs\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.125334 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.125252 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-metrics-certs\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.125334 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.125300 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx84r\" (UniqueName: \"kubernetes.io/projected/e2094290-5903-4a68-bbf7-78734a56887f-kube-api-access-vx84r\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.125334 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.125331 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2094290-5903-4a68-bbf7-78734a56887f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.125554 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.125382 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.126088 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.126043 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2094290-5903-4a68-bbf7-78734a56887f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.127706 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.127670 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.127859 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.127834 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-metrics-certs\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.133366 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.133338 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx84r\" (UniqueName: \"kubernetes.io/projected/e2094290-5903-4a68-bbf7-78734a56887f-kube-api-access-vx84r\") pod \"maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.180875 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.180845 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:40.459201 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.459169 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf"] Sep 07 04:12:40.461258 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:12:40.461232 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2094290_5903_4a68_bbf7_78734a56887f.slice/crio-3bd36f4bb0bba51b2244c9f7850c58f23911f0605e27aa1ffb96e0a33462ca13 WatchSource:0}: Error finding container 3bd36f4bb0bba51b2244c9f7850c58f23911f0605e27aa1ffb96e0a33462ca13: Status 404 returned error can't find the container with id 3bd36f4bb0bba51b2244c9f7850c58f23911f0605e27aa1ffb96e0a33462ca13 Sep 07 04:12:40.777549 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:40.777409 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" event={"ID":"e2094290-5903-4a68-bbf7-78734a56887f","Type":"ContainerStarted","Data":"3bd36f4bb0bba51b2244c9f7850c58f23911f0605e27aa1ffb96e0a33462ca13"} Sep 07 04:12:41.784124 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:41.784084 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" event={"ID":"e2094290-5903-4a68-bbf7-78734a56887f","Type":"ContainerStarted","Data":"e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e"} Sep 07 04:12:41.784519 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:41.784188 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:12:41.801006 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:41.800956 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" podStartSLOduration=2.279491082 podStartE2EDuration="2.80094249s" totalImagesPullingTime="521.451408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:12:41.80029001 +0000 UTC m=+1480.375156299" watchObservedRunningTime="2026-09-07 04:12:41.80094249 +0000 UTC m=+1480.375808779" Sep 07 04:12:44.590114 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:44.590072 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:12:47.796325 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:12:47.796297 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:13:02.011619 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:02.011591 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:13:02.013501 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:02.013480 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:13:02.193403 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:02.193374 2555 scope.go:117] "RemoveContainer" containerID="6da848b64030a62bbe2f1edc98ab8f4b6c7eb65149fb29c223ada7e1de80e332" Sep 07 04:13:02.201717 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:02.201697 2555 scope.go:117] "RemoveContainer" containerID="73a9a24ca01e12762a7122c466d738daeb5c43ec344cf3fe4937c28d6adbef07" Sep 07 04:13:02.209517 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:02.209495 2555 scope.go:117] "RemoveContainer" containerID="30f5eb9c78dcca4db7170a2fc454ac56cc7de700dd9fbef19557617e31920054" Sep 07 04:13:16.295712 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:16.295673 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:13:28.192456 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:28.192374 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:13:35.039154 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.039116 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q"] Sep 07 04:13:35.043383 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.043354 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.045756 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.045735 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-c5ce6a-openshift-default-dockercfg-b7lv2\"" Sep 07 04:13:35.054186 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.054160 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q"] Sep 07 04:13:35.149681 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.149643 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-data\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.149681 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.149692 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-token\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.149950 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.149734 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-certs\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.149950 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.149766 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139dcc12-a622-480b-8a7a-e0bfed215c1f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.149950 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.149791 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-envoy\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.149950 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.149842 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swqq\" (UniqueName: \"kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-kube-api-access-9swqq\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.149950 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.149909 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-socket\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.149950 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.149945 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-podinfo\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.150299 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.149972 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-credential-socket\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.250769 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.250733 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-socket\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.250769 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.250775 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-podinfo\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251017 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.250793 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-credential-socket\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251017 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.250919 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-data\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251017 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.250951 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-token\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251017 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.251002 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-certs\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.251038 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139dcc12-a622-480b-8a7a-e0bfed215c1f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.251091 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-envoy\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.251161 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swqq\" (UniqueName: \"kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-kube-api-access-9swqq\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.251206 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-credential-socket\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.251271 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-socket\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251553 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.251293 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-data\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251553 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.251350 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-certs\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.251882 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.251860 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139dcc12-a622-480b-8a7a-e0bfed215c1f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.253262 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.253237 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-podinfo\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.253262 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.253247 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-envoy\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.258014 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.257995 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-token\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.258540 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.258519 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swqq\" (UniqueName: \"kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-kube-api-access-9swqq\") pod \"e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.356627 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.356546 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:35.631121 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.631095 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q"] Sep 07 04:13:35.632967 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:13:35.632940 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139dcc12_a622_480b_8a7a_e0bfed215c1f.slice/crio-da6437d25330941c545e88063dfbd7c21bc02511c5954de0dfb9cf2142d79403 WatchSource:0}: Error finding container da6437d25330941c545e88063dfbd7c21bc02511c5954de0dfb9cf2142d79403: Status 404 returned error can't find the container with id da6437d25330941c545e88063dfbd7c21bc02511c5954de0dfb9cf2142d79403 Sep 07 04:13:35.636003 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.635965 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:13:35.636113 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.636083 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:13:35.636161 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:35.636146 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:13:36.067586 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:36.067546 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" event={"ID":"139dcc12-a622-480b-8a7a-e0bfed215c1f","Type":"ContainerStarted","Data":"4071f88b022a457d2d25c4545531d6157f96f58e090deee753a67d42e5e1c10f"} Sep 07 04:13:36.067946 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:36.067608 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" event={"ID":"139dcc12-a622-480b-8a7a-e0bfed215c1f","Type":"ContainerStarted","Data":"da6437d25330941c545e88063dfbd7c21bc02511c5954de0dfb9cf2142d79403"} Sep 07 04:13:36.087114 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:36.087067 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" podStartSLOduration=1.087039097 podStartE2EDuration="1.087039097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:13:36.084294235 +0000 UTC m=+1534.659160523" watchObservedRunningTime="2026-09-07 04:13:36.087039097 +0000 UTC m=+1534.661905387" Sep 07 04:13:36.357652 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:36.357565 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:36.363165 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:36.363140 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:37.073119 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:37.073089 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:37.074163 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:37.074141 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:42.488440 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:42.488409 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:13:53.115203 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:53.115162 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 04:13:53.115562 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:53.115243 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 04:13:54.170002 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:54.169896 2555 generic.go:356] "Generic (PLEG): container finished" podID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerID="4071f88b022a457d2d25c4545531d6157f96f58e090deee753a67d42e5e1c10f" exitCode=0 Sep 07 04:13:54.170002 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:54.169975 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" event={"ID":"139dcc12-a622-480b-8a7a-e0bfed215c1f","Type":"ContainerDied","Data":"4071f88b022a457d2d25c4545531d6157f96f58e090deee753a67d42e5e1c10f"} Sep 07 04:13:54.170581 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:54.170550 2555 scope.go:117] "RemoveContainer" containerID="4071f88b022a457d2d25c4545531d6157f96f58e090deee753a67d42e5e1c10f" Sep 07 04:13:54.171789 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:54.171756 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:13:54.171886 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:54.171828 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:13:54.171886 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:54.171865 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 07 04:13:55.177140 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:55.177105 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" event={"ID":"139dcc12-a622-480b-8a7a-e0bfed215c1f","Type":"ContainerStarted","Data":"0bf9a1b30c54f9ace340c04f4e24c795a24b42f30531f23988eebb040c585795"} Sep 07 04:13:55.357466 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:55.357418 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:13:55.357675 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:55.357607 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 07 04:13:55.357757 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:55.357667 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 07 04:13:56.357575 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:56.357537 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 07 04:13:56.358183 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:56.358148 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 07 04:13:57.357448 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:57.357411 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 07 04:13:57.357639 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:57.357473 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 07 04:13:58.357121 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:58.357076 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 07 04:13:58.357308 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:58.357161 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 07 04:13:59.357182 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:59.357122 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 07 04:13:59.357708 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:13:59.357191 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 07 04:14:00.360443 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:00.360407 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:14:00.360890 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:00.360681 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:14:00.361398 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:00.361381 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:14:14.785361 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:14.785320 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:14:16.131926 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:16.131895 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q"] Sep 07 04:14:16.132417 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:16.132157 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" containerID="cri-o://0bf9a1b30c54f9ace340c04f4e24c795a24b42f30531f23988eebb040c585795" gracePeriod=30 Sep 07 04:14:21.312962 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.312923 2555 generic.go:356] "Generic (PLEG): container finished" podID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerID="0bf9a1b30c54f9ace340c04f4e24c795a24b42f30531f23988eebb040c585795" exitCode=0 Sep 07 04:14:21.313414 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.312993 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" event={"ID":"139dcc12-a622-480b-8a7a-e0bfed215c1f","Type":"ContainerDied","Data":"0bf9a1b30c54f9ace340c04f4e24c795a24b42f30531f23988eebb040c585795"} Sep 07 04:14:21.313414 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.313084 2555 scope.go:117] "RemoveContainer" containerID="4071f88b022a457d2d25c4545531d6157f96f58e090deee753a67d42e5e1c10f" Sep 07 04:14:21.391131 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.391101 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:14:21.507690 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.507655 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-data\") pod \"139dcc12-a622-480b-8a7a-e0bfed215c1f\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " Sep 07 04:14:21.507690 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.507691 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-envoy\") pod \"139dcc12-a622-480b-8a7a-e0bfed215c1f\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " Sep 07 04:14:21.507964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.507715 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-socket\") pod \"139dcc12-a622-480b-8a7a-e0bfed215c1f\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " Sep 07 04:14:21.507964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.507762 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-credential-socket\") pod \"139dcc12-a622-480b-8a7a-e0bfed215c1f\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " Sep 07 04:14:21.507964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.507781 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-certs\") pod \"139dcc12-a622-480b-8a7a-e0bfed215c1f\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " Sep 07 04:14:21.507964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.507829 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-token\") pod \"139dcc12-a622-480b-8a7a-e0bfed215c1f\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " Sep 07 04:14:21.507964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.507859 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9swqq\" (UniqueName: \"kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-kube-api-access-9swqq\") pod \"139dcc12-a622-480b-8a7a-e0bfed215c1f\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " Sep 07 04:14:21.507964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.507888 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139dcc12-a622-480b-8a7a-e0bfed215c1f-istiod-ca-cert\") pod \"139dcc12-a622-480b-8a7a-e0bfed215c1f\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " Sep 07 04:14:21.507964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.507928 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-podinfo\") pod \"139dcc12-a622-480b-8a7a-e0bfed215c1f\" (UID: \"139dcc12-a622-480b-8a7a-e0bfed215c1f\") " Sep 07 04:14:21.508361 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.508020 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "139dcc12-a622-480b-8a7a-e0bfed215c1f" (UID: "139dcc12-a622-480b-8a7a-e0bfed215c1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:14:21.508361 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.508047 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "139dcc12-a622-480b-8a7a-e0bfed215c1f" (UID: "139dcc12-a622-480b-8a7a-e0bfed215c1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:14:21.508361 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.508081 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "139dcc12-a622-480b-8a7a-e0bfed215c1f" (UID: "139dcc12-a622-480b-8a7a-e0bfed215c1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:14:21.508361 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.508307 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139dcc12-a622-480b-8a7a-e0bfed215c1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "139dcc12-a622-480b-8a7a-e0bfed215c1f" (UID: "139dcc12-a622-480b-8a7a-e0bfed215c1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:14:21.508361 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.508339 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:14:21.508736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.508366 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:14:21.508736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.508382 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:14:21.508736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.508486 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "139dcc12-a622-480b-8a7a-e0bfed215c1f" (UID: "139dcc12-a622-480b-8a7a-e0bfed215c1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:14:21.510047 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.510025 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "139dcc12-a622-480b-8a7a-e0bfed215c1f" (UID: "139dcc12-a622-480b-8a7a-e0bfed215c1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:14:21.510159 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.510119 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "139dcc12-a622-480b-8a7a-e0bfed215c1f" (UID: "139dcc12-a622-480b-8a7a-e0bfed215c1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:14:21.510424 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.510407 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-kube-api-access-9swqq" (OuterVolumeSpecName: "kube-api-access-9swqq") pod "139dcc12-a622-480b-8a7a-e0bfed215c1f" (UID: "139dcc12-a622-480b-8a7a-e0bfed215c1f"). InnerVolumeSpecName "kube-api-access-9swqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:14:21.510484 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.510438 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "139dcc12-a622-480b-8a7a-e0bfed215c1f" (UID: "139dcc12-a622-480b-8a7a-e0bfed215c1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:14:21.609889 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.609853 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:14:21.609889 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.609885 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:14:21.609889 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.609894 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9swqq\" (UniqueName: \"kubernetes.io/projected/139dcc12-a622-480b-8a7a-e0bfed215c1f-kube-api-access-9swqq\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:14:21.610148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.609904 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139dcc12-a622-480b-8a7a-e0bfed215c1f-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:14:21.610148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.609913 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:14:21.610148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:21.609921 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139dcc12-a622-480b-8a7a-e0bfed215c1f-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:14:22.319625 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:22.319589 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" event={"ID":"139dcc12-a622-480b-8a7a-e0bfed215c1f","Type":"ContainerDied","Data":"da6437d25330941c545e88063dfbd7c21bc02511c5954de0dfb9cf2142d79403"} Sep 07 04:14:22.319625 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:22.319622 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q" Sep 07 04:14:22.320174 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:22.319652 2555 scope.go:117] "RemoveContainer" containerID="0bf9a1b30c54f9ace340c04f4e24c795a24b42f30531f23988eebb040c585795" Sep 07 04:14:22.337455 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:22.337430 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q"] Sep 07 04:14:22.339241 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:22.339224 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c5ce6a-openshift-default-988b56dcd-pst9q"] Sep 07 04:14:23.938745 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:23.938702 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" path="/var/lib/kubelet/pods/139dcc12-a622-480b-8a7a-e0bfed215c1f/volumes" Sep 07 04:14:33.490168 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:33.490133 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:14:45.089478 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:14:45.089441 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:15:29.660904 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:29.660866 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4"] Sep 07 04:15:29.661440 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:29.661167 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" containerID="cri-o://3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483" gracePeriod=30 Sep 07 04:15:32.309702 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:32.309660 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 04:15:32.310125 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:32.309724 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 04:15:34.923454 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:34.923388 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:15:35.011721 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.011683 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-data\") pod \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " Sep 07 04:15:35.011896 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.011748 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-token\") pod \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " Sep 07 04:15:35.011896 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.011822 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-podinfo\") pod \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " Sep 07 04:15:35.011896 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.011854 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-envoy\") pod \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " Sep 07 04:15:35.011896 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.011881 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-socket\") pod \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " Sep 07 04:15:35.012148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.011956 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istiod-ca-cert\") pod \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " Sep 07 04:15:35.012148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.011997 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-credential-socket\") pod \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " Sep 07 04:15:35.012148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.012022 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-certs\") pod \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " Sep 07 04:15:35.012148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.012035 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f9cfa32-08d4-4107-bd1b-60f1a927e471" (UID: "2f9cfa32-08d4-4107-bd1b-60f1a927e471"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:15:35.012148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.012101 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4kpq\" (UniqueName: \"kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-kube-api-access-m4kpq\") pod \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\" (UID: \"2f9cfa32-08d4-4107-bd1b-60f1a927e471\") " Sep 07 04:15:35.012404 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.012290 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f9cfa32-08d4-4107-bd1b-60f1a927e471" (UID: "2f9cfa32-08d4-4107-bd1b-60f1a927e471"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:15:35.012466 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.012456 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:35.012526 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.012463 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f9cfa32-08d4-4107-bd1b-60f1a927e471" (UID: "2f9cfa32-08d4-4107-bd1b-60f1a927e471"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:15:35.012526 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.012474 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:35.012526 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.012496 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f9cfa32-08d4-4107-bd1b-60f1a927e471" (UID: "2f9cfa32-08d4-4107-bd1b-60f1a927e471"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:15:35.012698 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.012662 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f9cfa32-08d4-4107-bd1b-60f1a927e471" (UID: "2f9cfa32-08d4-4107-bd1b-60f1a927e471"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:15:35.014039 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.014014 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f9cfa32-08d4-4107-bd1b-60f1a927e471" (UID: "2f9cfa32-08d4-4107-bd1b-60f1a927e471"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:15:35.014181 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.014082 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f9cfa32-08d4-4107-bd1b-60f1a927e471" (UID: "2f9cfa32-08d4-4107-bd1b-60f1a927e471"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:15:35.014278 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.014190 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f9cfa32-08d4-4107-bd1b-60f1a927e471" (UID: "2f9cfa32-08d4-4107-bd1b-60f1a927e471"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:15:35.014586 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.014566 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-kube-api-access-m4kpq" (OuterVolumeSpecName: "kube-api-access-m4kpq") pod "2f9cfa32-08d4-4107-bd1b-60f1a927e471" (UID: "2f9cfa32-08d4-4107-bd1b-60f1a927e471"). InnerVolumeSpecName "kube-api-access-m4kpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:15:35.113266 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.113231 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:35.113266 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.113261 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:35.113266 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.113270 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:35.113508 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.113279 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:35.113508 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.113289 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9cfa32-08d4-4107-bd1b-60f1a927e471-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:35.113508 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.113300 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4kpq\" (UniqueName: \"kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-kube-api-access-m4kpq\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:35.113508 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.113308 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9cfa32-08d4-4107-bd1b-60f1a927e471-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:35.708766 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.708735 2555 generic.go:356] "Generic (PLEG): container finished" podID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerID="3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483" exitCode=0 Sep 07 04:15:35.708964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.708779 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" Sep 07 04:15:35.708964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.708817 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" event={"ID":"2f9cfa32-08d4-4107-bd1b-60f1a927e471","Type":"ContainerDied","Data":"3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483"} Sep 07 04:15:35.708964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.708884 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4" event={"ID":"2f9cfa32-08d4-4107-bd1b-60f1a927e471","Type":"ContainerDied","Data":"8ce766caba6bdf766050beadc60f2c6da8a61e9cd13c42d7582f413879bc4576"} Sep 07 04:15:35.708964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.708913 2555 scope.go:117] "RemoveContainer" containerID="3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483" Sep 07 04:15:35.718967 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.718951 2555 scope.go:117] "RemoveContainer" containerID="82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235" Sep 07 04:15:35.727894 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.727876 2555 scope.go:117] "RemoveContainer" containerID="3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483" Sep 07 04:15:35.728273 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:15:35.728248 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483\": container with ID starting with 3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483 not found: ID does not exist" containerID="3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483" Sep 07 04:15:35.728375 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.728283 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483"} err="failed to get container status \"3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483\": rpc error: code = NotFound desc = could not find container \"3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483\": container with ID starting with 3d8600a09b8eec5ca63a2e6d8ec46f0ecff3a971a31811f24a543acef699a483 not found: ID does not exist" Sep 07 04:15:35.728375 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.728304 2555 scope.go:117] "RemoveContainer" containerID="82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235" Sep 07 04:15:35.728601 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:15:35.728582 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235\": container with ID starting with 82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235 not found: ID does not exist" containerID="82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235" Sep 07 04:15:35.728664 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.728606 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235"} err="failed to get container status \"82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235\": rpc error: code = NotFound desc = could not find container \"82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235\": container with ID starting with 82e3d984ed173604cc4cba28f71a40391903556e5e1b1e544fd8d3238ff6b235 not found: ID does not exist" Sep 07 04:15:35.730472 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.730449 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4"] Sep 07 04:15:35.733671 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.733652 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f8fdfa-openshift-default-5c78554dd4-nj9s4"] Sep 07 04:15:35.937672 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:35.937637 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" path="/var/lib/kubelet/pods/2f9cfa32-08d4-4107-bd1b-60f1a927e471/volumes" Sep 07 04:15:40.776154 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:40.776114 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf"] Sep 07 04:15:40.776659 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:40.776360 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" podUID="e2094290-5903-4a68-bbf7-78734a56887f" containerName="maas-api" containerID="cri-o://e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e" gracePeriod=30 Sep 07 04:15:41.029462 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.029399 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:15:41.170975 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.170943 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-metrics-certs\") pod \"e2094290-5903-4a68-bbf7-78734a56887f\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " Sep 07 04:15:41.171167 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.171035 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx84r\" (UniqueName: \"kubernetes.io/projected/e2094290-5903-4a68-bbf7-78734a56887f-kube-api-access-vx84r\") pod \"e2094290-5903-4a68-bbf7-78734a56887f\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " Sep 07 04:15:41.171167 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.171130 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-maas-api-tls\") pod \"e2094290-5903-4a68-bbf7-78734a56887f\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " Sep 07 04:15:41.171167 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.171165 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2094290-5903-4a68-bbf7-78734a56887f-openshift-service-ca\") pod \"e2094290-5903-4a68-bbf7-78734a56887f\" (UID: \"e2094290-5903-4a68-bbf7-78734a56887f\") " Sep 07 04:15:41.171591 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.171566 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2094290-5903-4a68-bbf7-78734a56887f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2094290-5903-4a68-bbf7-78734a56887f" (UID: "e2094290-5903-4a68-bbf7-78734a56887f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:15:41.173187 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.173160 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e2094290-5903-4a68-bbf7-78734a56887f" (UID: "e2094290-5903-4a68-bbf7-78734a56887f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:15:41.173282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.173179 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2094290-5903-4a68-bbf7-78734a56887f-kube-api-access-vx84r" (OuterVolumeSpecName: "kube-api-access-vx84r") pod "e2094290-5903-4a68-bbf7-78734a56887f" (UID: "e2094290-5903-4a68-bbf7-78734a56887f"). InnerVolumeSpecName "kube-api-access-vx84r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:15:41.173282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.173216 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2094290-5903-4a68-bbf7-78734a56887f" (UID: "e2094290-5903-4a68-bbf7-78734a56887f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:15:41.272561 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.272528 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-maas-api-tls\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:41.272561 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.272554 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2094290-5903-4a68-bbf7-78734a56887f-openshift-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:41.272561 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.272564 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2094290-5903-4a68-bbf7-78734a56887f-metrics-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:41.272561 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.272572 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx84r\" (UniqueName: \"kubernetes.io/projected/e2094290-5903-4a68-bbf7-78734a56887f-kube-api-access-vx84r\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:15:41.747697 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.747663 2555 generic.go:356] "Generic (PLEG): container finished" podID="e2094290-5903-4a68-bbf7-78734a56887f" containerID="e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e" exitCode=0 Sep 07 04:15:41.747905 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.747706 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" Sep 07 04:15:41.747905 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.747743 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" event={"ID":"e2094290-5903-4a68-bbf7-78734a56887f","Type":"ContainerDied","Data":"e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e"} Sep 07 04:15:41.747905 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.747803 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf" event={"ID":"e2094290-5903-4a68-bbf7-78734a56887f","Type":"ContainerDied","Data":"3bd36f4bb0bba51b2244c9f7850c58f23911f0605e27aa1ffb96e0a33462ca13"} Sep 07 04:15:41.747905 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.747833 2555 scope.go:117] "RemoveContainer" containerID="e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e" Sep 07 04:15:41.757540 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.757524 2555 scope.go:117] "RemoveContainer" containerID="e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e" Sep 07 04:15:41.757794 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:15:41.757775 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e\": container with ID starting with e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e not found: ID does not exist" containerID="e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e" Sep 07 04:15:41.757853 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.757806 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e"} err="failed to get container status \"e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e\": rpc error: code = NotFound desc = could not find container \"e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e\": container with ID starting with e6087b1e8809db6f62b55c710df2d4e0073c2ef9a96a9c229b370baa7acb856e not found: ID does not exist" Sep 07 04:15:41.771975 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.771945 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf"] Sep 07 04:15:41.775696 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.775671 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-704b16-8756cb57d-5hvqf"] Sep 07 04:15:41.937743 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:41.937708 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2094290-5903-4a68-bbf7-78734a56887f" path="/var/lib/kubelet/pods/e2094290-5903-4a68-bbf7-78734a56887f/volumes" Sep 07 04:15:44.685878 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:15:44.685850 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:16:00.496945 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:00.496908 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:16:04.809174 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:04.809136 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7"] Sep 07 04:16:04.809595 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:04.809443 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" containerID="cri-o://d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1" gracePeriod=30 Sep 07 04:16:10.060093 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.060052 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:16:10.135942 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.135862 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-podinfo\") pod \"b6ee7e09-c93d-49fb-a581-4523b390964a\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " Sep 07 04:16:10.135942 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.135894 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-certs\") pod \"b6ee7e09-c93d-49fb-a581-4523b390964a\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " Sep 07 04:16:10.135942 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.135930 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-token\") pod \"b6ee7e09-c93d-49fb-a581-4523b390964a\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " Sep 07 04:16:10.136250 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.135958 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hckqh\" (UniqueName: \"kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-kube-api-access-hckqh\") pod \"b6ee7e09-c93d-49fb-a581-4523b390964a\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " Sep 07 04:16:10.136250 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.135985 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ee7e09-c93d-49fb-a581-4523b390964a-istiod-ca-cert\") pod \"b6ee7e09-c93d-49fb-a581-4523b390964a\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " Sep 07 04:16:10.136250 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.136040 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-credential-socket\") pod \"b6ee7e09-c93d-49fb-a581-4523b390964a\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " Sep 07 04:16:10.136250 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.136083 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-envoy\") pod \"b6ee7e09-c93d-49fb-a581-4523b390964a\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " Sep 07 04:16:10.136250 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.136112 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-socket\") pod \"b6ee7e09-c93d-49fb-a581-4523b390964a\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " Sep 07 04:16:10.136250 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.136155 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-data\") pod \"b6ee7e09-c93d-49fb-a581-4523b390964a\" (UID: \"b6ee7e09-c93d-49fb-a581-4523b390964a\") " Sep 07 04:16:10.136250 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.136183 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6ee7e09-c93d-49fb-a581-4523b390964a" (UID: "b6ee7e09-c93d-49fb-a581-4523b390964a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:16:10.136665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.136472 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:10.136665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.136539 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6ee7e09-c93d-49fb-a581-4523b390964a" (UID: "b6ee7e09-c93d-49fb-a581-4523b390964a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:16:10.136665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.136613 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6ee7e09-c93d-49fb-a581-4523b390964a" (UID: "b6ee7e09-c93d-49fb-a581-4523b390964a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:16:10.136665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.136649 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ee7e09-c93d-49fb-a581-4523b390964a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6ee7e09-c93d-49fb-a581-4523b390964a" (UID: "b6ee7e09-c93d-49fb-a581-4523b390964a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:16:10.136665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.136661 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6ee7e09-c93d-49fb-a581-4523b390964a" (UID: "b6ee7e09-c93d-49fb-a581-4523b390964a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:16:10.138261 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.138227 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6ee7e09-c93d-49fb-a581-4523b390964a" (UID: "b6ee7e09-c93d-49fb-a581-4523b390964a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 04:16:10.138458 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.138435 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-kube-api-access-hckqh" (OuterVolumeSpecName: "kube-api-access-hckqh") pod "b6ee7e09-c93d-49fb-a581-4523b390964a" (UID: "b6ee7e09-c93d-49fb-a581-4523b390964a"). InnerVolumeSpecName "kube-api-access-hckqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:16:10.138635 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.138616 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6ee7e09-c93d-49fb-a581-4523b390964a" (UID: "b6ee7e09-c93d-49fb-a581-4523b390964a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:16:10.138717 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.138691 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6ee7e09-c93d-49fb-a581-4523b390964a" (UID: "b6ee7e09-c93d-49fb-a581-4523b390964a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:16:10.237480 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.237447 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-token\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:10.237480 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.237478 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hckqh\" (UniqueName: \"kubernetes.io/projected/b6ee7e09-c93d-49fb-a581-4523b390964a-kube-api-access-hckqh\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:10.237657 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.237491 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ee7e09-c93d-49fb-a581-4523b390964a-istiod-ca-cert\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:10.237657 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.237500 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-credential-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:10.237657 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.237508 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-envoy\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:10.237657 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.237517 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-workload-socket\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:10.237657 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.237526 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-data\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:10.237657 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.237533 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ee7e09-c93d-49fb-a581-4523b390964a-istio-podinfo\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:10.901561 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.901529 2555 generic.go:356] "Generic (PLEG): container finished" podID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerID="d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1" exitCode=0 Sep 07 04:16:10.901749 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.901566 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" event={"ID":"b6ee7e09-c93d-49fb-a581-4523b390964a","Type":"ContainerDied","Data":"d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1"} Sep 07 04:16:10.901749 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.901589 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" Sep 07 04:16:10.901749 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.901647 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7" event={"ID":"b6ee7e09-c93d-49fb-a581-4523b390964a","Type":"ContainerDied","Data":"569c55529fab0b6cb123b3a15eea0ee141b96b50e694a0f4519fa184ef99e5c0"} Sep 07 04:16:10.901749 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.901671 2555 scope.go:117] "RemoveContainer" containerID="d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1" Sep 07 04:16:10.910974 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.910959 2555 scope.go:117] "RemoveContainer" containerID="e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227" Sep 07 04:16:10.919134 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.919118 2555 scope.go:117] "RemoveContainer" containerID="d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1" Sep 07 04:16:10.919394 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:16:10.919373 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1\": container with ID starting with d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1 not found: ID does not exist" containerID="d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1" Sep 07 04:16:10.919449 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.919402 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1"} err="failed to get container status \"d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1\": rpc error: code = NotFound desc = could not find container \"d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1\": container with ID starting with d304b0dd90309c5e3853229c727dd90f6f66626834665e19c3afba2b659a26e1 not found: ID does not exist" Sep 07 04:16:10.919449 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.919417 2555 scope.go:117] "RemoveContainer" containerID="e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227" Sep 07 04:16:10.919645 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:16:10.919627 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227\": container with ID starting with e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227 not found: ID does not exist" containerID="e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227" Sep 07 04:16:10.919705 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.919653 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227"} err="failed to get container status \"e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227\": rpc error: code = NotFound desc = could not find container \"e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227\": container with ID starting with e6128ff4ee0effbb0778f839923461a226beda50dceefe8f950af8ec0f20d227 not found: ID does not exist" Sep 07 04:16:10.923523 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.923502 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7"] Sep 07 04:16:10.926913 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:10.926893 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8cea6-openshift-default-bb4946cd-vlpx7"] Sep 07 04:16:11.936914 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:11.936883 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" path="/var/lib/kubelet/pods/b6ee7e09-c93d-49fb-a581-4523b390964a/volumes" Sep 07 04:16:15.534874 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.534840 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9"] Sep 07 04:16:15.535277 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.535134 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" podUID="3f0fa4d5-1a1b-4523-93a5-66208522223d" containerName="maas-api" containerID="cri-o://d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603" gracePeriod=30 Sep 07 04:16:15.777033 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.777009 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:16:15.892314 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.892226 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-maas-api-tls\") pod \"3f0fa4d5-1a1b-4523-93a5-66208522223d\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " Sep 07 04:16:15.892314 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.892306 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qvwn\" (UniqueName: \"kubernetes.io/projected/3f0fa4d5-1a1b-4523-93a5-66208522223d-kube-api-access-7qvwn\") pod \"3f0fa4d5-1a1b-4523-93a5-66208522223d\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " Sep 07 04:16:15.892547 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.892378 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-metrics-certs\") pod \"3f0fa4d5-1a1b-4523-93a5-66208522223d\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " Sep 07 04:16:15.892547 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.892404 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0fa4d5-1a1b-4523-93a5-66208522223d-openshift-service-ca\") pod \"3f0fa4d5-1a1b-4523-93a5-66208522223d\" (UID: \"3f0fa4d5-1a1b-4523-93a5-66208522223d\") " Sep 07 04:16:15.892823 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.892796 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0fa4d5-1a1b-4523-93a5-66208522223d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f0fa4d5-1a1b-4523-93a5-66208522223d" (UID: "3f0fa4d5-1a1b-4523-93a5-66208522223d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:16:15.894335 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.894313 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f0fa4d5-1a1b-4523-93a5-66208522223d" (UID: "3f0fa4d5-1a1b-4523-93a5-66208522223d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:16:15.894447 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.894360 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f0fa4d5-1a1b-4523-93a5-66208522223d" (UID: "3f0fa4d5-1a1b-4523-93a5-66208522223d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:16:15.894447 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.894412 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0fa4d5-1a1b-4523-93a5-66208522223d-kube-api-access-7qvwn" (OuterVolumeSpecName: "kube-api-access-7qvwn") pod "3f0fa4d5-1a1b-4523-93a5-66208522223d" (UID: "3f0fa4d5-1a1b-4523-93a5-66208522223d"). InnerVolumeSpecName "kube-api-access-7qvwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:16:15.929997 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.929966 2555 generic.go:356] "Generic (PLEG): container finished" podID="3f0fa4d5-1a1b-4523-93a5-66208522223d" containerID="d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603" exitCode=0 Sep 07 04:16:15.930139 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.930075 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" Sep 07 04:16:15.935913 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.935885 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" event={"ID":"3f0fa4d5-1a1b-4523-93a5-66208522223d","Type":"ContainerDied","Data":"d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603"} Sep 07 04:16:15.936080 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.935927 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9" event={"ID":"3f0fa4d5-1a1b-4523-93a5-66208522223d","Type":"ContainerDied","Data":"6cbc61be6a2c35fe52fa482f32f8fededdad73ff047b535c2d5f5bbed693a25f"} Sep 07 04:16:15.936080 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.935950 2555 scope.go:117] "RemoveContainer" containerID="d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603" Sep 07 04:16:15.950684 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.950658 2555 scope.go:117] "RemoveContainer" containerID="d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603" Sep 07 04:16:15.951008 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:16:15.950986 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603\": container with ID starting with d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603 not found: ID does not exist" containerID="d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603" Sep 07 04:16:15.951111 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.951014 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603"} err="failed to get container status \"d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603\": rpc error: code = NotFound desc = could not find container \"d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603\": container with ID starting with d0f7d3a2a16c405f360682a2d20e2f48159d2de7e493b6fe87e9d49537895603 not found: ID does not exist" Sep 07 04:16:15.961512 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.961489 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9"] Sep 07 04:16:15.965797 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.965779 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434d57-65798c9b5b-gkmp9"] Sep 07 04:16:15.993153 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.993129 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-maas-api-tls\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:15.993153 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.993150 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qvwn\" (UniqueName: \"kubernetes.io/projected/3f0fa4d5-1a1b-4523-93a5-66208522223d-kube-api-access-7qvwn\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:15.993272 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.993159 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0fa4d5-1a1b-4523-93a5-66208522223d-metrics-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:15.993272 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:15.993168 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0fa4d5-1a1b-4523-93a5-66208522223d-openshift-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:16:17.936714 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:17.936680 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f0fa4d5-1a1b-4523-93a5-66208522223d" path="/var/lib/kubelet/pods/3f0fa4d5-1a1b-4523-93a5-66208522223d/volumes" Sep 07 04:16:18.188582 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:18.188507 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:16:28.188787 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:16:28.188708 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:17:02.275432 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:17:02.275397 2555 scope.go:117] "RemoveContainer" containerID="fac7b5ed2f5f6fa530548edb0029abc3d91fa257c2c73dc8ae842a678b48ba19" Sep 07 04:17:02.283356 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:17:02.283334 2555 scope.go:117] "RemoveContainer" containerID="33c465f901ab27297a98a9adde8286e313544bbff466cd465b80e75810ad81e7" Sep 07 04:17:47.288398 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:17:47.288364 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:18:02.041282 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:18:02.041248 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:18:02.044212 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:18:02.044190 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:18:14.290480 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:18:14.290445 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:18:22.191357 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:18:22.191315 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:18:29.181984 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:18:29.181947 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:18:38.382773 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:18:38.382734 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:18:51.683094 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:18:51.683050 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:18:59.390112 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:18:59.390075 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:19:05.011956 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:05.011920 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk"] Sep 07 04:19:05.012365 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:05.012127 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" podUID="176fc359-0dc1-4fbe-854e-9e7daccb381b" containerName="manager" containerID="cri-o://986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd" gracePeriod=10 Sep 07 04:19:06.263681 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.263657 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 04:19:06.293661 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.292997 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2k72\" (UniqueName: \"kubernetes.io/projected/176fc359-0dc1-4fbe-854e-9e7daccb381b-kube-api-access-s2k72\") pod \"176fc359-0dc1-4fbe-854e-9e7daccb381b\" (UID: \"176fc359-0dc1-4fbe-854e-9e7daccb381b\") " Sep 07 04:19:06.293661 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.293164 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176fc359-0dc1-4fbe-854e-9e7daccb381b-extensions-socket-volume\") pod \"176fc359-0dc1-4fbe-854e-9e7daccb381b\" (UID: \"176fc359-0dc1-4fbe-854e-9e7daccb381b\") " Sep 07 04:19:06.293941 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.293685 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176fc359-0dc1-4fbe-854e-9e7daccb381b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "176fc359-0dc1-4fbe-854e-9e7daccb381b" (UID: "176fc359-0dc1-4fbe-854e-9e7daccb381b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 04:19:06.296731 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.296702 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176fc359-0dc1-4fbe-854e-9e7daccb381b-kube-api-access-s2k72" (OuterVolumeSpecName: "kube-api-access-s2k72") pod "176fc359-0dc1-4fbe-854e-9e7daccb381b" (UID: "176fc359-0dc1-4fbe-854e-9e7daccb381b"). InnerVolumeSpecName "kube-api-access-s2k72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:19:06.394031 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.394000 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2k72\" (UniqueName: \"kubernetes.io/projected/176fc359-0dc1-4fbe-854e-9e7daccb381b-kube-api-access-s2k72\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:19:06.394031 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.394028 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176fc359-0dc1-4fbe-854e-9e7daccb381b-extensions-socket-volume\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:19:06.746840 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.746802 2555 generic.go:356] "Generic (PLEG): container finished" podID="176fc359-0dc1-4fbe-854e-9e7daccb381b" containerID="986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd" exitCode=0 Sep 07 04:19:06.747011 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.746859 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" Sep 07 04:19:06.747011 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.746880 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" event={"ID":"176fc359-0dc1-4fbe-854e-9e7daccb381b","Type":"ContainerDied","Data":"986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd"} Sep 07 04:19:06.747011 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.746963 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk" event={"ID":"176fc359-0dc1-4fbe-854e-9e7daccb381b","Type":"ContainerDied","Data":"8f8c8c99e39d875cf89a05995a705acdb8a22ae0d8a431588d24effadb5dbdc5"} Sep 07 04:19:06.747011 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.746986 2555 scope.go:117] "RemoveContainer" containerID="986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd" Sep 07 04:19:06.757855 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.757834 2555 scope.go:117] "RemoveContainer" containerID="986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd" Sep 07 04:19:06.758191 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:19:06.758170 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd\": container with ID starting with 986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd not found: ID does not exist" containerID="986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd" Sep 07 04:19:06.758278 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.758202 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd"} err="failed to get container status \"986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd\": rpc error: code = NotFound desc = could not find container \"986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd\": container with ID starting with 986965604cebe4a55342e22b0bf376da3cad8255db1e5c4bd9264afec3baf1dd not found: ID does not exist" Sep 07 04:19:06.768264 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.768243 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk"] Sep 07 04:19:06.773393 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:06.773373 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-7vrpk"] Sep 07 04:19:07.936964 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:19:07.936933 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="176fc359-0dc1-4fbe-854e-9e7daccb381b" path="/var/lib/kubelet/pods/176fc359-0dc1-4fbe-854e-9e7daccb381b/volumes" Sep 07 04:20:05.701132 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701094 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq"] Sep 07 04:20:05.701534 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701485 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176fc359-0dc1-4fbe-854e-9e7daccb381b" containerName="manager" Sep 07 04:20:05.701534 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701498 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="176fc359-0dc1-4fbe-854e-9e7daccb381b" containerName="manager" Sep 07 04:20:05.701534 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701508 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" Sep 07 04:20:05.701534 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701513 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" Sep 07 04:20:05.701534 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701526 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" Sep 07 04:20:05.701534 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701532 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701546 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701551 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701559 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0fa4d5-1a1b-4523-93a5-66208522223d" containerName="maas-api" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701563 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0fa4d5-1a1b-4523-93a5-66208522223d" containerName="maas-api" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701570 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701575 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701585 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2094290-5903-4a68-bbf7-78734a56887f" containerName="maas-api" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701589 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2094290-5903-4a68-bbf7-78734a56887f" containerName="maas-api" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701636 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701643 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701650 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2094290-5903-4a68-bbf7-78734a56887f" containerName="maas-api" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701656 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701661 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701666 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0fa4d5-1a1b-4523-93a5-66208522223d" containerName="maas-api" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701671 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="176fc359-0dc1-4fbe-854e-9e7daccb381b" containerName="manager" Sep 07 04:20:05.701736 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.701676 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" Sep 07 04:20:05.704566 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.704550 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" Sep 07 04:20:05.707224 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.707204 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lst6j\"" Sep 07 04:20:05.715344 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.715323 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq"] Sep 07 04:20:05.787349 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.787320 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brxql\" (UniqueName: \"kubernetes.io/projected/baebffa5-98bb-47ae-b019-16f2ce91cf4e-kube-api-access-brxql\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-c8hvq\" (UID: \"baebffa5-98bb-47ae-b019-16f2ce91cf4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" Sep 07 04:20:05.787470 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.787363 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/baebffa5-98bb-47ae-b019-16f2ce91cf4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-c8hvq\" (UID: \"baebffa5-98bb-47ae-b019-16f2ce91cf4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" Sep 07 04:20:05.888240 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.888209 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brxql\" (UniqueName: \"kubernetes.io/projected/baebffa5-98bb-47ae-b019-16f2ce91cf4e-kube-api-access-brxql\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-c8hvq\" (UID: \"baebffa5-98bb-47ae-b019-16f2ce91cf4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" Sep 07 04:20:05.888386 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.888273 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/baebffa5-98bb-47ae-b019-16f2ce91cf4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-c8hvq\" (UID: \"baebffa5-98bb-47ae-b019-16f2ce91cf4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" Sep 07 04:20:05.888658 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.888638 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/baebffa5-98bb-47ae-b019-16f2ce91cf4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-c8hvq\" (UID: \"baebffa5-98bb-47ae-b019-16f2ce91cf4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" Sep 07 04:20:05.897052 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:05.897028 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brxql\" (UniqueName: \"kubernetes.io/projected/baebffa5-98bb-47ae-b019-16f2ce91cf4e-kube-api-access-brxql\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-c8hvq\" (UID: \"baebffa5-98bb-47ae-b019-16f2ce91cf4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" Sep 07 04:20:06.014917 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:06.014839 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" Sep 07 04:20:06.288169 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:06.288146 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq"] Sep 07 04:20:06.290979 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:20:06.290950 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaebffa5_98bb_47ae_b019_16f2ce91cf4e.slice/crio-c906e440161ad4b893a8fd68ea7b13aeea7aab3997a97b68a0e4f69bf4e1dbcd WatchSource:0}: Error finding container c906e440161ad4b893a8fd68ea7b13aeea7aab3997a97b68a0e4f69bf4e1dbcd: Status 404 returned error can't find the container with id c906e440161ad4b893a8fd68ea7b13aeea7aab3997a97b68a0e4f69bf4e1dbcd Sep 07 04:20:07.043366 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:07.043331 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" event={"ID":"baebffa5-98bb-47ae-b019-16f2ce91cf4e","Type":"ContainerStarted","Data":"f34983a989c509fae89042c1d1ab70252a962984e154a1568f07fb2dbf0b5f6c"} Sep 07 04:20:07.043809 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:07.043392 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" event={"ID":"baebffa5-98bb-47ae-b019-16f2ce91cf4e","Type":"ContainerStarted","Data":"c906e440161ad4b893a8fd68ea7b13aeea7aab3997a97b68a0e4f69bf4e1dbcd"} Sep 07 04:20:07.043809 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:07.043422 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" Sep 07 04:20:07.067851 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:07.067804 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" podStartSLOduration=2.067790986 podStartE2EDuration="2.067790986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:20:07.064852695 +0000 UTC m=+1925.639718982" watchObservedRunningTime="2026-09-07 04:20:07.067790986 +0000 UTC m=+1925.642657275" Sep 07 04:20:18.049704 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:18.049674 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-c8hvq" Sep 07 04:20:27.788336 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:27.788306 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:20:35.489775 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:35.489731 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:20:43.191149 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:43.191107 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:20:48.483159 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:20:48.483121 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:21:19.895504 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:21:19.895416 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:22:13.285544 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:13.285501 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:22:20.985442 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:20.985406 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:22:29.297385 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.297289 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw"] Sep 07 04:22:29.297830 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.297811 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" Sep 07 04:22:29.297830 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.297831 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9cfa32-08d4-4107-bd1b-60f1a927e471" containerName="istio-proxy" Sep 07 04:22:29.297949 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.297855 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" Sep 07 04:22:29.297949 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.297864 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="139dcc12-a622-480b-8a7a-e0bfed215c1f" containerName="istio-proxy" Sep 07 04:22:29.298072 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.297974 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6ee7e09-c93d-49fb-a581-4523b390964a" containerName="istio-proxy" Sep 07 04:22:29.301238 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.301219 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.304686 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.304655 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 07 04:22:29.304819 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.304688 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 04:22:29.304819 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.304708 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-m9mvz\"" Sep 07 04:22:29.304819 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.304712 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 07 04:22:29.304819 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.304688 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 07 04:22:29.310896 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.310871 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw"] Sep 07 04:22:29.369473 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.369440 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pprl\" (UniqueName: \"kubernetes.io/projected/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-kube-api-access-6pprl\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.369660 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.369502 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-maas-api-tls\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.369660 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.369565 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-metrics-certs\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.369660 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.369600 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-openshift-service-ca\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.470987 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.470950 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-maas-api-tls\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.470987 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.470992 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-metrics-certs\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.471280 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.471014 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-openshift-service-ca\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.471280 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.471105 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pprl\" (UniqueName: \"kubernetes.io/projected/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-kube-api-access-6pprl\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.471769 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.471746 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-openshift-service-ca\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.473469 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.473444 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-metrics-certs\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.473608 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.473586 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-maas-api-tls\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.478402 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.478374 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pprl\" (UniqueName: \"kubernetes.io/projected/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-kube-api-access-6pprl\") pod \"maas-api-86d6f5487f-lf2jw\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.612497 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.612415 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:29.882347 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.882316 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw"] Sep 07 04:22:29.885532 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:22:29.885505 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a0fddb_dbdf_48a5_93d9_9bfe94ade954.slice/crio-9a625f96ce2332c4aaaff878d25f2ee2fde22947627963050fd9467b923a9a45 WatchSource:0}: Error finding container 9a625f96ce2332c4aaaff878d25f2ee2fde22947627963050fd9467b923a9a45: Status 404 returned error can't find the container with id 9a625f96ce2332c4aaaff878d25f2ee2fde22947627963050fd9467b923a9a45 Sep 07 04:22:29.885532 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.885529 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:22:29.886878 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:29.886859 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 04:22:30.744359 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:30.744321 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" event={"ID":"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954","Type":"ContainerStarted","Data":"527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5"} Sep 07 04:22:30.744897 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:30.744417 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" event={"ID":"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954","Type":"ContainerStarted","Data":"9a625f96ce2332c4aaaff878d25f2ee2fde22947627963050fd9467b923a9a45"} Sep 07 04:22:30.744897 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:30.744442 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:30.764694 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:30.764629 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" podStartSLOduration=1.368879785 podStartE2EDuration="1.764612661s" totalImagesPullingTime="395.732876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:22:30.761719693 +0000 UTC m=+2069.336585995" watchObservedRunningTime="2026-09-07 04:22:30.764612661 +0000 UTC m=+2069.339478947" Sep 07 04:22:36.188597 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:36.188559 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:22:36.756163 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:36.756129 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:36.827118 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:36.827084 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm"] Sep 07 04:22:36.831288 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:36.831264 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:36.839757 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:36.839728 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm"] Sep 07 04:22:36.943388 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:36.943355 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fb081d-5d23-4b12-a81b-6da90be7f86e-openshift-service-ca\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:36.943573 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:36.943407 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62fb081d-5d23-4b12-a81b-6da90be7f86e-metrics-certs\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:36.943573 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:36.943433 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62fb081d-5d23-4b12-a81b-6da90be7f86e-maas-api-tls\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:36.943573 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:36.943449 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd8z9\" (UniqueName: \"kubernetes.io/projected/62fb081d-5d23-4b12-a81b-6da90be7f86e-kube-api-access-dd8z9\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:37.044443 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.044329 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fb081d-5d23-4b12-a81b-6da90be7f86e-openshift-service-ca\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:37.044443 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.044415 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62fb081d-5d23-4b12-a81b-6da90be7f86e-metrics-certs\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:37.044694 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.044450 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62fb081d-5d23-4b12-a81b-6da90be7f86e-maas-api-tls\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:37.044694 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.044478 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd8z9\" (UniqueName: \"kubernetes.io/projected/62fb081d-5d23-4b12-a81b-6da90be7f86e-kube-api-access-dd8z9\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:37.045226 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.045200 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fb081d-5d23-4b12-a81b-6da90be7f86e-openshift-service-ca\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:37.059321 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.059263 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62fb081d-5d23-4b12-a81b-6da90be7f86e-metrics-certs\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:37.059762 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.059583 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62fb081d-5d23-4b12-a81b-6da90be7f86e-maas-api-tls\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:37.062292 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.062269 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd8z9\" (UniqueName: \"kubernetes.io/projected/62fb081d-5d23-4b12-a81b-6da90be7f86e-kube-api-access-dd8z9\") pod \"maas-api-577644cfc5-7p9zm\" (UID: \"62fb081d-5d23-4b12-a81b-6da90be7f86e\") " pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:37.146414 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.146365 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:37.436966 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.436874 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm"] Sep 07 04:22:37.440568 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:22:37.440538 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fb081d_5d23_4b12_a81b_6da90be7f86e.slice/crio-15faffbb4d6ab305c0ecfc28cb3e72558eaa7241d4c0861db5c49fd0f580b6df WatchSource:0}: Error finding container 15faffbb4d6ab305c0ecfc28cb3e72558eaa7241d4c0861db5c49fd0f580b6df: Status 404 returned error can't find the container with id 15faffbb4d6ab305c0ecfc28cb3e72558eaa7241d4c0861db5c49fd0f580b6df Sep 07 04:22:37.780519 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:37.780483 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" event={"ID":"62fb081d-5d23-4b12-a81b-6da90be7f86e","Type":"ContainerStarted","Data":"15faffbb4d6ab305c0ecfc28cb3e72558eaa7241d4c0861db5c49fd0f580b6df"} Sep 07 04:22:38.787215 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:38.787172 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" event={"ID":"62fb081d-5d23-4b12-a81b-6da90be7f86e","Type":"ContainerStarted","Data":"8c2bb9f4d1f49b6e2ab1a27e87f8dbfffdad6e8c0a6b032c0a181a54a9dee952"} Sep 07 04:22:38.787601 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:38.787367 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:38.804890 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:38.804843 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" podStartSLOduration=2.426731474 podStartE2EDuration="2.804827451s" totalImagesPullingTime="378.095977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 04:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:22:38.802467661 +0000 UTC m=+2077.377333952" watchObservedRunningTime="2026-09-07 04:22:38.804827451 +0000 UTC m=+2077.379693736" Sep 07 04:22:44.798356 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:44.798323 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-577644cfc5-7p9zm" Sep 07 04:22:44.843114 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:44.843075 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw"] Sep 07 04:22:44.843351 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:44.843326 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" podUID="c9a0fddb-dbdf-48a5-93d9-9bfe94ade954" containerName="maas-api" containerID="cri-o://527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5" gracePeriod=30 Sep 07 04:22:45.282617 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:45.282579 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:22:46.095424 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.095392 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:46.235770 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.235738 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-maas-api-tls\") pod \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " Sep 07 04:22:46.235952 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.235817 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-metrics-certs\") pod \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " Sep 07 04:22:46.235952 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.235890 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-openshift-service-ca\") pod \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " Sep 07 04:22:46.236131 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.236104 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pprl\" (UniqueName: \"kubernetes.io/projected/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-kube-api-access-6pprl\") pod \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\" (UID: \"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954\") " Sep 07 04:22:46.236277 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.236254 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9a0fddb-dbdf-48a5-93d9-9bfe94ade954" (UID: "c9a0fddb-dbdf-48a5-93d9-9bfe94ade954"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 04:22:46.236486 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.236464 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-openshift-service-ca\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:22:46.238048 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.238022 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9a0fddb-dbdf-48a5-93d9-9bfe94ade954" (UID: "c9a0fddb-dbdf-48a5-93d9-9bfe94ade954"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:22:46.238171 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.238049 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9a0fddb-dbdf-48a5-93d9-9bfe94ade954" (UID: "c9a0fddb-dbdf-48a5-93d9-9bfe94ade954"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 04:22:46.238171 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.238156 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-kube-api-access-6pprl" (OuterVolumeSpecName: "kube-api-access-6pprl") pod "c9a0fddb-dbdf-48a5-93d9-9bfe94ade954" (UID: "c9a0fddb-dbdf-48a5-93d9-9bfe94ade954"). InnerVolumeSpecName "kube-api-access-6pprl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 04:22:46.337795 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.337758 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-metrics-certs\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:22:46.337795 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.337791 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pprl\" (UniqueName: \"kubernetes.io/projected/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-kube-api-access-6pprl\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:22:46.337795 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.337800 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954-maas-api-tls\") on node \"ip-10-0-131-215.ec2.internal\" DevicePath \"\"" Sep 07 04:22:46.827159 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.827125 2555 generic.go:356] "Generic (PLEG): container finished" podID="c9a0fddb-dbdf-48a5-93d9-9bfe94ade954" containerID="527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5" exitCode=0 Sep 07 04:22:46.827369 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.827178 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" Sep 07 04:22:46.827369 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.827213 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" event={"ID":"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954","Type":"ContainerDied","Data":"527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5"} Sep 07 04:22:46.827369 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.827266 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw" event={"ID":"c9a0fddb-dbdf-48a5-93d9-9bfe94ade954","Type":"ContainerDied","Data":"9a625f96ce2332c4aaaff878d25f2ee2fde22947627963050fd9467b923a9a45"} Sep 07 04:22:46.827369 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.827289 2555 scope.go:117] "RemoveContainer" containerID="527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5" Sep 07 04:22:46.836869 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.836845 2555 scope.go:117] "RemoveContainer" containerID="527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5" Sep 07 04:22:46.837164 ip-10-0-131-215 kubenswrapper[2555]: E0907 04:22:46.837144 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5\": container with ID starting with 527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5 not found: ID does not exist" containerID="527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5" Sep 07 04:22:46.837247 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.837172 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5"} err="failed to get container status \"527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5\": rpc error: code = NotFound desc = could not find container \"527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5\": container with ID starting with 527dc75f5760686dc43107898922e2fb1692ceb926aba109253ec5d0193be1e5 not found: ID does not exist" Sep 07 04:22:46.849655 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.849628 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw"] Sep 07 04:22:46.853369 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:46.853339 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d6f5487f-lf2jw"] Sep 07 04:22:47.937128 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:47.937086 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a0fddb-dbdf-48a5-93d9-9bfe94ade954" path="/var/lib/kubelet/pods/c9a0fddb-dbdf-48a5-93d9-9bfe94ade954/volumes" Sep 07 04:22:52.896229 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:52.896193 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:22:59.794017 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:22:59.793979 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7tl54"] Sep 07 04:23:02.071782 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:02.071753 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:23:02.076831 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:02.076806 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:23:02.923386 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:02.923352 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7bd9996b56-wm7mx_53e18245-6542-4222-8d9a-ced383830854/authorino/0.log" Sep 07 04:23:08.229134 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:08.229084 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-44pf7_91c7d218-7958-4f12-9ddc-b06d91c091b8/manager/0.log" Sep 07 04:23:08.923471 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:08.923443 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756ff4ddd-r5gvs_e6dc622d-3312-4719-956d-401de581ccc7/copy-manifests/0.log" Sep 07 04:23:08.929177 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:08.929154 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756ff4ddd-r5gvs_e6dc622d-3312-4719-956d-401de581ccc7/manager/0.log" Sep 07 04:23:09.410374 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:09.410345 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-577644cfc5-7p9zm_62fb081d-5d23-4b12-a81b-6da90be7f86e/maas-api/0.log" Sep 07 04:23:10.573183 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:10.573142 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7bd9996b56-wm7mx_53e18245-6542-4222-8d9a-ced383830854/authorino/0.log" Sep 07 04:23:10.703320 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:10.703294 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-xcn2k_c6a5a2bc-b46d-45ac-9dae-559d9fa2de80/manager/0.log" Sep 07 04:23:11.036034 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:11.036007 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-c8hvq_baebffa5-98bb-47ae-b019-16f2ce91cf4e/manager/0.log" Sep 07 04:23:11.169294 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:11.169265 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-7tl54_79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8/limitador/0.log" Sep 07 04:23:11.285619 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:11.285591 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-g28g2_91846c5b-9098-450e-82ba-e782fe2b1c74/manager/0.log" Sep 07 04:23:11.981639 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:11.981609 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59d7d88d64-pvnz2_b78d4100-aaa6-4b75-9804-2a7d5faa86c3/kube-auth-proxy/0.log" Sep 07 04:23:12.097981 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:12.097952 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-d7pgb_ab9bed40-0504-4031-b120-d3d199e315a6/istio-proxy/1.log" Sep 07 04:23:12.466203 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:12.466176 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7447dd5b74-p7pm5_cc36ab08-3dc2-4011-b24a-4ee2635c4630/router/0.log" Sep 07 04:23:13.037677 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:13.037649 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-rtv4n_8215cbec-b876-475d-b35f-afa868f2d6ef/main/0.log" Sep 07 04:23:13.148394 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:13.148364 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ddwhv_ff86904a-3699-4173-bbf2-7786d666ff9f/main/0.log" Sep 07 04:23:13.258519 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:13.258489 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcpg9d_97920ef3-0c7c-438b-9508-3d3b0ddd0c92/main/0.log" Sep 07 04:23:21.628255 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:21.628224 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gbx7k_4a784083-f7eb-4508-9ef5-07ae8c51a973/global-pull-secret-syncer/0.log" Sep 07 04:23:21.773115 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:21.773084 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hp2z7_f834b816-08bb-466b-8b19-d1af19636f85/konnectivity-agent/0.log" Sep 07 04:23:21.830790 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:21.830768 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-215.ec2.internal_bed84f1ba21f69b974e5fd2991bf7d19/haproxy/0.log" Sep 07 04:23:26.458965 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:26.458935 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7bd9996b56-wm7mx_53e18245-6542-4222-8d9a-ced383830854/authorino/0.log" Sep 07 04:23:26.493955 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:26.493928 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-xcn2k_c6a5a2bc-b46d-45ac-9dae-559d9fa2de80/manager/0.log" Sep 07 04:23:26.597240 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:26.597158 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-c8hvq_baebffa5-98bb-47ae-b019-16f2ce91cf4e/manager/0.log" Sep 07 04:23:26.622394 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:26.622372 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-7tl54_79d04ba1-10bd-4364-ace4-ecd7ecc1a3f8/limitador/0.log" Sep 07 04:23:26.673620 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:26.673596 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-g28g2_91846c5b-9098-450e-82ba-e782fe2b1c74/manager/0.log" Sep 07 04:23:28.370947 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:28.370921 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-5qh6c_8b8f75fd-b50c-461a-9f2f-d8433861d39a/cluster-monitoring-operator/0.log" Sep 07 04:23:28.607809 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:28.607785 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f6k55_c18d70f7-9e5a-4ab5-902a-c19904a894f7/node-exporter/0.log" Sep 07 04:23:28.636196 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:28.636135 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f6k55_c18d70f7-9e5a-4ab5-902a-c19904a894f7/kube-rbac-proxy/0.log" Sep 07 04:23:28.655140 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:28.655117 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f6k55_c18d70f7-9e5a-4ab5-902a-c19904a894f7/init-textfile/0.log" Sep 07 04:23:30.741574 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:30.741544 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-2j9wk_3d9cf8b6-a25a-4e43-9299-aba1b9ddf7d6/networking-console-plugin/0.log" Sep 07 04:23:31.294879 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.294848 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/1.log" Sep 07 04:23:31.299431 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.299407 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zvrg9_577e74b9-769c-48fa-b70f-e87314e9f179/console-operator/2.log" Sep 07 04:23:31.526556 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.526521 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh"] Sep 07 04:23:31.527017 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.526997 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a0fddb-dbdf-48a5-93d9-9bfe94ade954" containerName="maas-api" Sep 07 04:23:31.527163 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.527074 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a0fddb-dbdf-48a5-93d9-9bfe94ade954" containerName="maas-api" Sep 07 04:23:31.527226 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.527183 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a0fddb-dbdf-48a5-93d9-9bfe94ade954" containerName="maas-api" Sep 07 04:23:31.530469 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.530449 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.532713 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.532695 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nrrkw\"/\"default-dockercfg-tlzxz\"" Sep 07 04:23:31.533612 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.533595 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nrrkw\"/\"kube-root-ca.crt\"" Sep 07 04:23:31.533705 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.533618 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nrrkw\"/\"openshift-service-ca.crt\"" Sep 07 04:23:31.539480 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.539459 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh"] Sep 07 04:23:31.621429 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.621376 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-lib-modules\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.621429 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.621405 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bfbj\" (UniqueName: \"kubernetes.io/projected/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-kube-api-access-6bfbj\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.621550 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.621465 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-proc\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.621550 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.621507 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-sys\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.621550 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.621531 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-podres\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.722152 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.722132 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-lib-modules\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.722245 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.722158 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfbj\" (UniqueName: \"kubernetes.io/projected/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-kube-api-access-6bfbj\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.722245 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.722187 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-proc\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.722318 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.722242 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-sys\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.722318 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.722257 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-proc\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.722318 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.722282 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-podres\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.722318 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.722295 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-lib-modules\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.722318 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.722309 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-sys\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.722483 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.722387 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-podres\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.729849 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.729832 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bfbj\" (UniqueName: \"kubernetes.io/projected/0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2-kube-api-access-6bfbj\") pod \"perf-node-gather-daemonset-6hbgh\" (UID: \"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2\") " pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:31.840633 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:31.840608 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:32.109192 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:32.109169 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh"] Sep 07 04:23:32.111437 ip-10-0-131-215 kubenswrapper[2555]: W0907 04:23:32.111412 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0f2b7b7b_ecf8_4989_a99c_2fcd4267c4e2.slice/crio-ae06c19a87230821c0ccc970547314a68fe63559ab2d4ed1275fab163ecdd305 WatchSource:0}: Error finding container ae06c19a87230821c0ccc970547314a68fe63559ab2d4ed1275fab163ecdd305: Status 404 returned error can't find the container with id ae06c19a87230821c0ccc970547314a68fe63559ab2d4ed1275fab163ecdd305 Sep 07 04:23:32.579195 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:32.579168 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-ltc2c_6ea44a63-9e02-4390-b39a-d849c618285a/volume-data-source-validator/0.log" Sep 07 04:23:33.058279 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:33.058248 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" event={"ID":"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2","Type":"ContainerStarted","Data":"053d9d57f7046a8aad6df5b51b76fb5c76bb8349ba12e2cf5a69005b4221fc04"} Sep 07 04:23:33.058665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:33.058295 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" event={"ID":"0f2b7b7b-ecf8-4989-a99c-2fcd4267c4e2","Type":"ContainerStarted","Data":"ae06c19a87230821c0ccc970547314a68fe63559ab2d4ed1275fab163ecdd305"} Sep 07 04:23:33.058665 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:33.058377 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:33.071861 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:33.071822 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" podStartSLOduration=2.071808912 podStartE2EDuration="2.071808912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 04:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 04:23:33.071391098 +0000 UTC m=+2131.646257387" watchObservedRunningTime="2026-09-07 04:23:33.071808912 +0000 UTC m=+2131.646675240" Sep 07 04:23:33.746148 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:33.746123 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rh982_08336dd4-dd4c-47e3-ba8d-531ca667a0b5/dns/0.log" Sep 07 04:23:33.763686 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:33.763660 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rh982_08336dd4-dd4c-47e3-ba8d-531ca667a0b5/kube-rbac-proxy/0.log" Sep 07 04:23:33.783226 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:33.783206 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6wvfr_b83fecd8-da21-48bf-a8f4-13c33d46742a/dns-node-resolver/0.log" Sep 07 04:23:34.364433 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:34.364401 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fst8d_ca804fd3-f751-4640-8e5a-a831ab9e2f55/node-ca/0.log" Sep 07 04:23:35.921482 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:35.921452 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59d7d88d64-pvnz2_b78d4100-aaa6-4b75-9804-2a7d5faa86c3/kube-auth-proxy/0.log" Sep 07 04:23:35.944695 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:35.944666 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-d7pgb_ab9bed40-0504-4031-b120-d3d199e315a6/istio-proxy/0.log" Sep 07 04:23:35.953556 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:35.953536 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-d7pgb_ab9bed40-0504-4031-b120-d3d199e315a6/istio-proxy/1.log" Sep 07 04:23:36.056358 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:36.056337 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7447dd5b74-p7pm5_cc36ab08-3dc2-4011-b24a-4ee2635c4630/router/0.log" Sep 07 04:23:36.907971 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:36.907943 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gljg7_212b6720-24d5-46df-8053-6859f2e790ca/serve-healthcheck-canary/0.log" Sep 07 04:23:37.392492 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:37.392463 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-bml8f_ed5e0dcd-3686-4f2d-8710-36f47ade6997/insights-operator/0.log" Sep 07 04:23:37.394176 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:37.394157 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-bml8f_ed5e0dcd-3686-4f2d-8710-36f47ade6997/insights-operator/1.log" Sep 07 04:23:37.537879 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:37.537843 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wv9z9_dd7656b6-1017-4966-92af-fd1d32d08a7f/kube-rbac-proxy/0.log" Sep 07 04:23:37.558422 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:37.558389 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wv9z9_dd7656b6-1017-4966-92af-fd1d32d08a7f/exporter/0.log" Sep 07 04:23:37.577065 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:37.577041 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wv9z9_dd7656b6-1017-4966-92af-fd1d32d08a7f/extractor/0.log" Sep 07 04:23:39.072619 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:39.072596 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nrrkw/perf-node-gather-daemonset-6hbgh" Sep 07 04:23:39.765817 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:39.765788 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-44pf7_91c7d218-7958-4f12-9ddc-b06d91c091b8/manager/0.log" Sep 07 04:23:40.213884 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:40.213857 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756ff4ddd-r5gvs_e6dc622d-3312-4719-956d-401de581ccc7/manager/0.log" Sep 07 04:23:40.238861 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:40.238840 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756ff4ddd-r5gvs_e6dc622d-3312-4719-956d-401de581ccc7/copy-manifests/0.log" Sep 07 04:23:42.804361 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:42.804288 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59d7d88d64-pvnz2_b78d4100-aaa6-4b75-9804-2a7d5faa86c3/kube-auth-proxy/0.log" Sep 07 04:23:42.828865 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:42.828836 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-d7pgb_ab9bed40-0504-4031-b120-d3d199e315a6/istio-proxy/0.log" Sep 07 04:23:42.837421 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:42.837400 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-d7pgb_ab9bed40-0504-4031-b120-d3d199e315a6/istio-proxy/1.log" Sep 07 04:23:42.918430 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:42.918406 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7447dd5b74-p7pm5_cc36ab08-3dc2-4011-b24a-4ee2635c4630/router/0.log" Sep 07 04:23:44.943545 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:44.943516 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59d7d88d64-pvnz2_b78d4100-aaa6-4b75-9804-2a7d5faa86c3/kube-auth-proxy/0.log" Sep 07 04:23:44.964396 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:44.964367 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-d7pgb_ab9bed40-0504-4031-b120-d3d199e315a6/istio-proxy/0.log" Sep 07 04:23:44.972818 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:44.972794 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-d7pgb_ab9bed40-0504-4031-b120-d3d199e315a6/istio-proxy/1.log" Sep 07 04:23:45.060920 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:45.060900 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7447dd5b74-p7pm5_cc36ab08-3dc2-4011-b24a-4ee2635c4630/router/0.log" Sep 07 04:23:46.886080 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:46.886033 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-vnd5t_10582104-f53f-40dd-9f07-81d404420cd4/migrator/0.log" Sep 07 04:23:46.902945 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:46.902920 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-vnd5t_10582104-f53f-40dd-9f07-81d404420cd4/graceful-termination/0.log" Sep 07 04:23:47.235400 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:47.235363 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-rmznn_4e43e9f6-5add-434b-89cc-d9db04f24dc7/kube-storage-version-migrator-operator/1.log" Sep 07 04:23:47.236382 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:47.236357 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-rmznn_4e43e9f6-5add-434b-89cc-d9db04f24dc7/kube-storage-version-migrator-operator/0.log" Sep 07 04:23:48.541247 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:48.541218 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcg5f_1231ac90-f33a-4adb-bfb4-8bd0b14c9348/kube-multus-additional-cni-plugins/0.log" Sep 07 04:23:48.561104 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:48.561074 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcg5f_1231ac90-f33a-4adb-bfb4-8bd0b14c9348/egress-router-binary-copy/0.log" Sep 07 04:23:48.579206 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:48.579181 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcg5f_1231ac90-f33a-4adb-bfb4-8bd0b14c9348/cni-plugins/0.log" Sep 07 04:23:48.596962 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:48.596944 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcg5f_1231ac90-f33a-4adb-bfb4-8bd0b14c9348/bond-cni-plugin/0.log" Sep 07 04:23:48.616535 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:48.616514 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcg5f_1231ac90-f33a-4adb-bfb4-8bd0b14c9348/routeoverride-cni/0.log" Sep 07 04:23:48.634522 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:48.634500 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcg5f_1231ac90-f33a-4adb-bfb4-8bd0b14c9348/whereabouts-cni-bincopy/0.log" Sep 07 04:23:48.652494 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:48.652472 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcg5f_1231ac90-f33a-4adb-bfb4-8bd0b14c9348/whereabouts-cni/0.log" Sep 07 04:23:48.694517 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:48.694493 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vg569_cd01acdb-295d-4cb0-ac8c-e0db0f5de8e9/kube-multus/0.log" Sep 07 04:23:48.803051 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:48.802944 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sxp6n_9d176a6b-c22a-4f6a-9551-1dbd25928cf8/network-metrics-daemon/0.log" Sep 07 04:23:48.819583 ip-10-0-131-215 kubenswrapper[2555]: I0907 04:23:48.819533 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sxp6n_9d176a6b-c22a-4f6a-9551-1dbd25928cf8/kube-rbac-proxy/0.log"