Jul 31 14:46:14.513637 ip-10-0-135-143 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 31 14:46:14.513651 ip-10-0-135-143 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 31 14:46:14.513660 ip-10-0-135-143 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 31 14:46:14.514098 ip-10-0-135-143 systemd[1]: Failed to start Kubernetes Kubelet. Jul 31 14:46:24.709828 ip-10-0-135-143 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 31 14:46:24.709847 ip-10-0-135-143 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f77f09a2530f4ed381167783ab437488 -- Jul 31 14:48:08.307614 ip-10-0-135-143 systemd[1]: Starting Kubernetes Kubelet... Jul 31 14:48:08.605062 ip-10-0-135-143 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 14:48:08.605062 ip-10-0-135-143 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 31 14:48:08.605062 ip-10-0-135-143 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 14:48:08.605062 ip-10-0-135-143 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 31 14:48:08.605062 ip-10-0-135-143 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 14:48:08.606462 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.606382 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 31 14:48:08.609593 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609574 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:48:08.609593 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609589 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:48:08.609593 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609594 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:48:08.609593 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609598 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609603 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609607 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609613 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609617 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609621 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609624 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609628 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609632 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609636 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609640 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609643 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609647 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609656 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609661 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609665 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609672 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609678 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609682 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:48:08.609815 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609687 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609691 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609695 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609699 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609703 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609709 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609713 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609718 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609722 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609726 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609731 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609735 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609739 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609744 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609748 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609753 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609757 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609763 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609767 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609772 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:48:08.610619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609776 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609780 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609784 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609789 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609793 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609798 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609803 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609808 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609812 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609816 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609820 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609824 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609828 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609833 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609838 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609842 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609846 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609851 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609855 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:48:08.611479 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609862 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609867 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609871 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609876 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609880 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609885 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609889 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609893 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609899 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609903 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609907 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609911 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609917 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609922 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609926 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609931 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609935 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609939 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609944 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:48:08.612295 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609950 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609954 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609958 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609962 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609966 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609970 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609975 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609979 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609985 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609990 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.609995 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.610000 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.610004 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.610009 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.610014 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.610019 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.610023 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.610027 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.610032 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611168 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:48:08.613131 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611178 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611183 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611188 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611192 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611196 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611201 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611205 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611209 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611213 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611218 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611222 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611226 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611231 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611236 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611241 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611245 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611249 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611253 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611257 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611261 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611265 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:48:08.613992 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611269 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611274 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611278 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611282 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611287 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611292 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611296 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611300 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611304 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611308 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611313 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611318 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611322 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611327 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611331 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611337 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611343 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611348 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611352 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:48:08.614636 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611356 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611361 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611365 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611371 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611375 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611379 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611384 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611388 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611393 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611397 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611401 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611405 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611410 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611415 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611420 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611424 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611428 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611432 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611436 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:48:08.615229 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611440 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611445 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611449 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611453 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611458 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611463 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611470 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611475 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611480 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611484 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611488 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611492 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611496 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611500 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611504 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611509 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611514 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611518 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611522 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611526 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:48:08.615959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611531 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611536 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611540 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611544 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611548 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611552 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611574 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611577 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611582 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611586 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611590 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611595 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611599 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611603 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611607 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611611 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611616 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611620 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.611624 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612181 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:48:08.616457 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612189 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612193 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612201 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612207 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612212 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612217 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612221 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612227 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612232 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612236 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612240 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612245 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612250 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612254 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612258 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612262 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612266 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612271 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612275 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:48:08.617041 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612279 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612283 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612287 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612291 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612295 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612299 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612303 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612306 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612310 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612315 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612319 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612323 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612327 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612332 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612336 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612340 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612344 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612349 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612353 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612358 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:48:08.617549 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612362 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612368 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612372 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612376 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612381 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612385 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612390 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612394 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612398 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612403 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612407 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612413 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612418 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612424 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612428 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612433 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612437 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612441 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612445 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:48:08.618368 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612449 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612454 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612458 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612462 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612467 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612471 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612475 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612479 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612483 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612488 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612492 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612496 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612500 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612504 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612508 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612513 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612517 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612521 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612525 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612530 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:48:08.619127 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612534 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612538 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612543 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612547 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612551 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612573 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612580 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612584 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612588 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612592 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612596 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612601 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612607 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612612 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612616 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612620 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612625 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612629 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612633 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.612638 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:48:08.619945 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612739 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612749 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612758 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612765 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612774 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612779 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612785 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612791 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612797 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612802 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612808 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612813 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612818 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612824 2582 flags.go:64] FLAG: --cgroup-root="" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612829 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612833 2582 flags.go:64] FLAG: --client-ca-file="" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612838 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612843 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612852 2582 flags.go:64] FLAG: --cluster-domain="" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612857 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612862 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612868 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612873 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612879 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 31 14:48:08.620619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612884 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612889 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612894 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612899 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612904 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612908 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612913 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612918 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612924 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612928 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612933 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612938 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612943 2582 flags.go:64] FLAG: --enable-server="true" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612947 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612953 2582 flags.go:64] FLAG: --event-burst="100" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612958 2582 flags.go:64] FLAG: --event-qps="50" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612962 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612967 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612972 2582 flags.go:64] FLAG: --eviction-hard="" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612978 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612983 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612987 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612993 2582 flags.go:64] FLAG: --eviction-soft="" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.612998 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613002 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 31 14:48:08.621193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613007 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613013 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613018 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613023 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613027 2582 flags.go:64] FLAG: --feature-gates="" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613033 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613038 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613042 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613047 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613052 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613059 2582 flags.go:64] FLAG: --help="false" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613064 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-135-143.ec2.internal" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613069 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613074 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613078 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613084 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613089 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613094 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613098 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613103 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613108 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613112 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613118 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613122 2582 flags.go:64] FLAG: --kube-reserved="" Jul 31 14:48:08.621852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613127 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613131 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613136 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613141 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613145 2582 flags.go:64] FLAG: --lock-file="" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613149 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613154 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613159 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613168 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613176 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613180 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613185 2582 flags.go:64] FLAG: --logging-format="text" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613189 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613195 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613200 2582 flags.go:64] FLAG: --manifest-url="" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613204 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613211 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613216 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613224 2582 flags.go:64] FLAG: --max-pods="110" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613229 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613234 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613238 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613243 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613248 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613252 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 31 14:48:08.622539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613257 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613268 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613272 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613277 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613282 2582 flags.go:64] FLAG: --pod-cidr="" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613286 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613295 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613299 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613304 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613309 2582 flags.go:64] FLAG: --port="10250" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613314 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613318 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-095d1d8258aa4b33a" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613323 2582 flags.go:64] FLAG: --qos-reserved="" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613328 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613334 2582 flags.go:64] FLAG: --register-node="true" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613339 2582 flags.go:64] FLAG: --register-with-taints="" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613344 2582 flags.go:64] FLAG: --registry-burst="10" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613351 2582 flags.go:64] FLAG: --registry-qps="5" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613355 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613360 2582 flags.go:64] FLAG: --reserved-memory="" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613366 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613371 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613376 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613381 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613386 2582 flags.go:64] FLAG: --runonce="false" Jul 31 14:48:08.623180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613392 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613397 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613401 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613406 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613410 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613415 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613420 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613425 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613430 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613435 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613439 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613444 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613449 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613453 2582 flags.go:64] FLAG: --system-cgroups="" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613458 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613466 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613471 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613476 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613482 2582 flags.go:64] FLAG: --tls-min-version="" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613487 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613491 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613496 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613501 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613506 2582 flags.go:64] FLAG: --v="2" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613514 2582 flags.go:64] FLAG: --version="false" Jul 31 14:48:08.623833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613519 2582 flags.go:64] FLAG: --vmodule="" Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613525 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.613530 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613716 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613726 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613732 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613737 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613745 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613750 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613754 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613759 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613763 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613768 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613772 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613777 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613781 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613785 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613789 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613793 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613799 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:48:08.624436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613804 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613809 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613813 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613817 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613821 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613825 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613829 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613834 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613838 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613843 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613848 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613854 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613858 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613862 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613866 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613870 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613874 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613878 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613883 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613889 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:48:08.624945 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613893 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613897 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613901 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613906 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613910 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613914 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613918 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613922 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613927 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613931 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613935 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613939 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613943 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613948 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613952 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613956 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613960 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613964 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613969 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613973 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:48:08.625436 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613977 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613981 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613986 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613992 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.613996 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614000 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614004 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614008 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614012 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614016 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614020 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614026 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614030 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614035 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614039 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614043 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614048 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614052 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614056 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614060 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:48:08.625935 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614064 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614068 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614073 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614080 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614086 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614091 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614095 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614100 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614105 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614109 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614113 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614117 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614122 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614126 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614130 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614136 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614141 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614145 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614149 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:48:08.626451 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614153 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614158 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.614162 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.614753 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.621873 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.621887 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.621906 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.621914 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.621966 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.621971 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.621974 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.621977 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.621981 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.621984 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:48:08.626932 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.621986 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.621990 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.621994 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.621998 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622001 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622004 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622006 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622009 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622012 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622014 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622017 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622019 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622022 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622025 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622027 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622030 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622033 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622035 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622037 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:48:08.627285 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622040 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622042 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622045 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622048 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622051 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622054 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622057 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622059 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622062 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622064 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622067 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622070 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622072 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622075 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622077 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622080 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622083 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622085 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622088 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622091 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:48:08.627761 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622093 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622095 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622098 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622101 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622103 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622106 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622108 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622111 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622113 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622116 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622118 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622120 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622123 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622125 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622128 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622131 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622134 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622137 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622140 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622142 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:48:08.628244 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622145 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622148 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622150 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622152 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622155 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622158 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622160 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622163 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622165 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622168 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622170 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622174 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622177 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622180 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622183 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622186 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622188 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622191 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622193 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:48:08.628830 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622196 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622198 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622201 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622203 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622206 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622209 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622211 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622214 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622218 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622222 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622225 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622229 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622232 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622234 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:48:08.629286 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622237 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.622242 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622344 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622349 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622353 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622357 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622360 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622363 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622366 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622369 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622371 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622374 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622377 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622380 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:48:08.629675 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622383 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622385 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622388 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622391 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622393 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622396 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622398 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622402 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622405 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622408 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622411 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622414 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622417 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622421 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622424 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622427 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622430 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622433 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622435 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622438 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:48:08.630025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622440 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622443 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622446 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622448 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622450 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622453 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622456 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622458 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622461 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622463 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622466 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622468 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622471 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622473 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622476 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622478 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622481 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622484 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622486 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622489 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:48:08.630502 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622491 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622493 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622496 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622498 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622501 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622504 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622506 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622510 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622513 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622516 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622519 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622522 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622524 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622527 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622529 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622532 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622535 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622537 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622540 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:48:08.631060 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622542 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622545 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622548 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622551 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622569 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622572 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622575 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622578 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622580 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622583 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622585 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622588 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622590 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622593 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622596 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622598 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622600 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622603 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622606 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622608 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:48:08.631803 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622611 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622613 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622616 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622619 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622621 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622624 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622626 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:08.622629 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.622634 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.622721 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.624777 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.625815 2582 server.go:1019] "Starting client certificate rotation" Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.625911 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 14:48:08.632289 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.625945 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 14:48:08.632705 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.627541 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 14:48:08.632705 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.632643 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 14:48:08.643963 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.643946 2582 log.go:25] "Validated CRI v1 runtime API" Jul 31 14:48:08.647991 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.647975 2582 log.go:25] "Validated CRI v1 image API" Jul 31 14:48:08.648836 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.648819 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 31 14:48:08.651726 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.651709 2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 963449f4-6578-49ca-83e1-4f0c0c57dfda:/dev/nvme0n1p4 ec4efae6-217f-4459-b6ca-a2d84498a5f8:/dev/nvme0n1p3] Jul 31 14:48:08.651785 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.651726 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 31 14:48:08.656833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.656726 2582 manager.go:216] Machine: {Timestamp:2026-07-31 14:48:08.655544982 +0000 UTC m=+0.256981310 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101391 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c106ab4649749bfcf2c869b2800da SystemUUID:ec2c106a-b464-9749-bfcf-2c869b2800da BootID:f77f09a2-530f-4ed3-8116-7783ab437488 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8d:9d:e2:f1:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8d:9d:e2:f1:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:3a:6d:ce:9d:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 31 14:48:08.656833 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.656825 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 31 14:48:08.656950 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.656920 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 31 14:48:08.658439 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.658424 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 14:48:08.659657 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.659632 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 31 14:48:08.659776 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.659659 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 31 14:48:08.660273 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.660259 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 31 14:48:08.660273 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.660271 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 31 14:48:08.660380 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.660278 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 14:48:08.660781 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.660770 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 14:48:08.661910 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.661899 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 31 14:48:08.665576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.665565 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 31 14:48:08.665799 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.665790 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 31 14:48:08.665835 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.665822 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 31 14:48:08.665865 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.665834 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 31 14:48:08.665865 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.665844 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 31 14:48:08.665865 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.665854 2582 kubelet.go:398] "Adding apiserver pod source" Jul 31 14:48:08.665865 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.665862 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 31 14:48:08.666159 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.666148 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 14:48:08.666201 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.666165 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 14:48:08.669079 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.668954 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 31 14:48:08.670396 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.670379 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 31 14:48:08.670458 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.670410 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 31 14:48:08.671636 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671625 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 31 14:48:08.671681 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671640 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 31 14:48:08.671681 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671646 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 31 14:48:08.671681 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671651 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 31 14:48:08.671681 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671660 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 31 14:48:08.671681 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671665 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 31 14:48:08.671681 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671675 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 31 14:48:08.671681 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671682 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 31 14:48:08.671864 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671689 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 31 14:48:08.671864 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671695 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 31 14:48:08.671864 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671703 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 31 14:48:08.671864 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.671708 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 31 14:48:08.672313 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.672303 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 31 14:48:08.672313 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.672312 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 31 14:48:08.675028 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.675009 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-143.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 31 14:48:08.675376 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.675364 2582 server.go:1268] "Started kubelet" Jul 31 14:48:08.675481 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.675433 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 31 14:48:08.675481 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.675437 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 31 14:48:08.675655 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.675457 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 31 14:48:08.675718 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.675494 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 31 14:48:08.675762 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.675529 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 31 14:48:08.676234 ip-10-0-135-143 systemd[1]: Started Kubernetes Kubelet. Jul 31 14:48:08.676857 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.676822 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 31 14:48:08.677428 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.677417 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 31 14:48:08.682095 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.682076 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 31 14:48:08.682158 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.681466 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18c7673ceea7617f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-31 14:48:08.675336575 +0000 UTC m=+0.276772908,LastTimestamp:2026-07-31 14:48:08.675336575 +0000 UTC m=+0.276772908,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}" Jul 31 14:48:08.683570 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.683541 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 31 14:48:08.683643 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.683573 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 31 14:48:08.683643 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.683619 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-46ltn" Jul 31 14:48:08.683720 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.683673 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:08.684190 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684173 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 31 14:48:08.684190 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684183 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 31 14:48:08.684190 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684192 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 31 14:48:08.684385 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684317 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 31 14:48:08.684385 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684325 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 31 14:48:08.684572 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684542 2582 factory.go:55] Registering systemd factory Jul 31 14:48:08.684572 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684573 2582 factory.go:223] Registration of the systemd container factory successfully Jul 31 14:48:08.684764 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684748 2582 factory.go:153] Registering CRI-O factory Jul 31 14:48:08.684764 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684767 2582 factory.go:223] Registration of the crio container factory successfully Jul 31 14:48:08.684886 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684817 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 31 14:48:08.684886 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684850 2582 factory.go:103] Registering Raw factory Jul 31 14:48:08.684886 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.684862 2582 manager.go:1196] Started watching for new ooms in manager Jul 31 14:48:08.685470 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.685456 2582 manager.go:349] Starting recovery of all containers Jul 31 14:48:08.687865 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.687815 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 31 14:48:08.688042 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.688013 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 31 14:48:08.693384 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.693359 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-46ltn" Jul 31 14:48:08.697701 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.697603 2582 manager.go:354] Recovery completed Jul 31 14:48:08.701783 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.701772 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:48:08.704634 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.704618 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:48:08.704707 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.704643 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:48:08.704707 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.704653 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:48:08.705090 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.705078 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 31 14:48:08.705090 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.705090 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 31 14:48:08.705173 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.705101 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:48:08.705173 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.705104 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 31 14:48:08.707120 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.707102 2582 policy_none.go:49] "None policy: Start" Jul 31 14:48:08.707120 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.707121 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 31 14:48:08.707249 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.707070 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18c7673cf0666325 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-31 14:48:08.704631589 +0000 UTC m=+0.306067917,LastTimestamp:2026-07-31 14:48:08.704631589 +0000 UTC m=+0.306067917,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}" Jul 31 14:48:08.707249 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.707138 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 31 14:48:08.707249 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.707230 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:48:08.707380 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.707253 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:48:08.707380 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.707268 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:48:08.709146 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.709135 2582 policy_none.go:47] "Start" Jul 31 14:48:08.739103 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.739087 2582 manager.go:337] "Starting Device Plugin manager" Jul 31 14:48:08.739175 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.739143 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 31 14:48:08.739175 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.739155 2582 server.go:87] "Starting device plugin registration server" Jul 31 14:48:08.739365 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.739352 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 31 14:48:08.739401 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.739369 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 31 14:48:08.739476 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.739459 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 31 14:48:08.739583 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.739549 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 31 14:48:08.739583 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.739577 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 31 14:48:08.739937 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.739916 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 31 14:48:08.740010 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.739968 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:08.767750 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.767724 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 31 14:48:08.780959 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.768901 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 31 14:48:08.780959 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.768934 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 31 14:48:08.780959 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.768952 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 31 14:48:08.780959 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.768981 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 31 14:48:08.780959 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.771439 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:48:08.839688 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.839672 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:48:08.840452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.840438 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:48:08.840535 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.840468 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:48:08.840535 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.840482 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:48:08.840535 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.840510 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-143.ec2.internal" Jul 31 14:48:08.848915 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.848900 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-143.ec2.internal" Jul 31 14:48:08.848998 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.848917 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-143.ec2.internal\": node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:08.867653 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.867635 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:08.869748 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.869730 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"] Jul 31 14:48:08.869820 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.869804 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:48:08.870723 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.870707 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:48:08.870791 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.870727 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:48:08.870791 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.870737 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:48:08.873118 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.873106 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:48:08.873248 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.873234 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" Jul 31 14:48:08.873303 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.873266 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:48:08.875204 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.875180 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:48:08.875290 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.875208 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:48:08.875290 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.875224 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:48:08.875290 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.875272 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:48:08.875421 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.875293 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:48:08.875421 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.875309 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:48:08.877590 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.877573 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" Jul 31 14:48:08.877706 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.877603 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:48:08.878244 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.878224 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:48:08.878329 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.878263 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:48:08.878329 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.878278 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:48:08.895179 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.895164 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-143.ec2.internal\" not found" node="ip-10-0-135-143.ec2.internal" Jul 31 14:48:08.899278 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.899263 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-143.ec2.internal\" not found" node="ip-10-0-135-143.ec2.internal" Jul 31 14:48:08.967923 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:08.967896 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:08.985867 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.985850 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/489ee1b4953de705708ca3f9e7a9eb02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"489ee1b4953de705708ca3f9e7a9eb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" Jul 31 14:48:08.985936 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.985876 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a155768c4417dbe455678d25197f0bfd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"a155768c4417dbe455678d25197f0bfd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" Jul 31 14:48:08.985936 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:08.985894 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/489ee1b4953de705708ca3f9e7a9eb02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"489ee1b4953de705708ca3f9e7a9eb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" Jul 31 14:48:09.069212 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:09.069178 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:09.086170 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.086148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/489ee1b4953de705708ca3f9e7a9eb02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"489ee1b4953de705708ca3f9e7a9eb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" Jul 31 14:48:09.086222 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.086182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/489ee1b4953de705708ca3f9e7a9eb02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"489ee1b4953de705708ca3f9e7a9eb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" Jul 31 14:48:09.086222 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.086201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a155768c4417dbe455678d25197f0bfd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"a155768c4417dbe455678d25197f0bfd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" Jul 31 14:48:09.086290 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.086222 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a155768c4417dbe455678d25197f0bfd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"a155768c4417dbe455678d25197f0bfd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" Jul 31 14:48:09.086290 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.086156 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/489ee1b4953de705708ca3f9e7a9eb02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"489ee1b4953de705708ca3f9e7a9eb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" Jul 31 14:48:09.086290 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.086279 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/489ee1b4953de705708ca3f9e7a9eb02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"489ee1b4953de705708ca3f9e7a9eb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" Jul 31 14:48:09.169801 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:09.169747 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:09.197199 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.197179 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" Jul 31 14:48:09.202867 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.202643 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" Jul 31 14:48:09.271585 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:09.271541 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:09.378107 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:09.378070 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:09.483111 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:09.483057 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:09.565407 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.565386 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:48:09.583980 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:09.583945 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:09.627686 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.627663 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 31 14:48:09.628261 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.627810 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 14:48:09.628261 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.627830 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 14:48:09.684489 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:09.684467 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:09.698392 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.698352 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 14:43:09 +0000 UTC" deadline="2028-02-13 06:19:39.980103001 +0000 UTC" Jul 31 14:48:09.698392 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.698386 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13479h31m30.281720439s" Jul 31 14:48:09.752579 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:09.752516 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:48:09.785485 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:09.785463 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:09.887373 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:09.887343 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:09.988411 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:09.988382 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:10.083693 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:10.083661 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda155768c4417dbe455678d25197f0bfd.slice/crio-7708c4e69c8eeacf0e73964e69a6d8ca8ec36705bec6efd18511e3fae1aee233 WatchSource:0}: Error finding container 7708c4e69c8eeacf0e73964e69a6d8ca8ec36705bec6efd18511e3fae1aee233: Status 404 returned error can't find the container with id 7708c4e69c8eeacf0e73964e69a6d8ca8ec36705bec6efd18511e3fae1aee233 Jul 31 14:48:10.087109 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.087091 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:48:10.088685 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.088666 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:10.189091 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.189069 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found" Jul 31 14:48:10.264095 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.263238 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:48:10.284880 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.284857 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" Jul 31 14:48:10.297403 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.297382 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 14:48:10.298203 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.298190 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" Jul 31 14:48:10.307701 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.307682 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 14:48:10.365649 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:10.365624 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489ee1b4953de705708ca3f9e7a9eb02.slice/crio-06237c5cc9e41c78aa4ddc718ba4baf4d101e8e595e4fcda3f7b789705c74413 WatchSource:0}: Error finding container 06237c5cc9e41c78aa4ddc718ba4baf4d101e8e595e4fcda3f7b789705c74413: Status 404 returned error can't find the container with id 06237c5cc9e41c78aa4ddc718ba4baf4d101e8e595e4fcda3f7b789705c74413 Jul 31 14:48:10.582600 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.582577 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:48:10.669255 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.669191 2582 apiserver.go:52] "Watching apiserver" Jul 31 14:48:10.676907 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.676885 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 31 14:48:10.677332 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.677300 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-tx9n9","kube-system/konnectivity-agent-b2xz8","kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal","openshift-cluster-node-tuning-operator/tuned-x9vd6","openshift-dns/node-resolver-tmrz2","openshift-image-registry/node-ca-s7km4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal","openshift-network-diagnostics/network-check-target-l4wr5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s","openshift-multus/multus-additional-cni-plugins-84fbb","openshift-multus/multus-jjklb","openshift-multus/network-metrics-daemon-thsf6","openshift-network-operator/iptables-alerter-zzp5r","openshift-ovn-kubernetes/ovnkube-node-2pdzx"] Jul 31 14:48:10.679998 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.679969 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.682599 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.682460 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:10.682706 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.682651 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 31 14:48:10.682854 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.682833 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 31 14:48:10.682938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.682910 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 31 14:48:10.683027 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.683003 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cqlt8\"" Jul 31 14:48:10.683126 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.683105 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 31 14:48:10.683180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.683119 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 31 14:48:10.684601 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.684568 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 31 14:48:10.684601 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.684582 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:48:10.684876 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.684858 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 31 14:48:10.685666 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.685311 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 31 14:48:10.685666 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.685318 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kqhlp\"" Jul 31 14:48:10.685666 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.685466 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.688288 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.688262 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 31 14:48:10.688865 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.688485 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xg67z\"" Jul 31 14:48:10.688865 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.688729 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 31 14:48:10.689642 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.689175 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 31 14:48:10.689642 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.689385 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 31 14:48:10.689642 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.689605 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 31 14:48:10.689859 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.689722 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 31 14:48:10.689859 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.689799 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.690383 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.690362 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:10.692586 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.692511 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tk9qt\"" Jul 31 14:48:10.692586 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.692537 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:48:10.693202 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.693178 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 31 14:48:10.693286 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.693268 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 31 14:48:10.693434 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.693403 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 31 14:48:10.694076 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.694050 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-49p62\"" Jul 31 14:48:10.694360 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.694343 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:10.695008 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.694984 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a356e358-2937-4638-8ae1-cb46daa39c33-iptables-alerter-script\") pod \"iptables-alerter-zzp5r\" (UID: \"a356e358-2937-4638-8ae1-cb46daa39c33\") " pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:10.695104 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695032 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vltd4\" (UniqueName: \"kubernetes.io/projected/a356e358-2937-4638-8ae1-cb46daa39c33-kube-api-access-vltd4\") pod \"iptables-alerter-zzp5r\" (UID: \"a356e358-2937-4638-8ae1-cb46daa39c33\") " pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:10.695104 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695068 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-kubelet\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695211 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695104 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-run-netns\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695211 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695139 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-etc-openvswitch\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695211 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-ovnkube-script-lib\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695411 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695235 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-cnibin\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.695411 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695269 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a356e358-2937-4638-8ae1-cb46daa39c33-host-slash\") pod \"iptables-alerter-zzp5r\" (UID: \"a356e358-2937-4638-8ae1-cb46daa39c33\") " pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:10.695411 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695297 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-run-systemd\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695411 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695328 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-env-overrides\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695411 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695365 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.695411 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695397 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-var-lib-openvswitch\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695428 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-node-log\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695490 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695558 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-log-socket\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695661 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-cni-bin\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.695712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695697 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-os-release\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.696007 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695736 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-slash\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.696007 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.696007 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695776 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-systemd-units\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.696007 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695795 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-ovnkube-config\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.696007 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.695939 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 14:48:10.696457 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.696435 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-ovn-node-metrics-cert\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.696540 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.696500 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-system-cni-dir\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.696540 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.696524 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 31 14:48:10.696648 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.696561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-cni-binary-copy\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.696725 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.696631 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.697028 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.697010 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.697120 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.697040 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s286j\" (UniqueName: \"kubernetes.io/projected/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-kube-api-access-s286j\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.697120 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.697089 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-run-ovn\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.697237 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.697152 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-run-openvswitch\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.697237 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.697195 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-cni-netd\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.697340 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.697260 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6rh\" (UniqueName: \"kubernetes.io/projected/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-kube-api-access-fw6rh\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.697575 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.697527 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 31 14:48:10.697936 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.697910 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qjsqz\"" Jul 31 14:48:10.698080 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.697917 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 31 14:48:10.699913 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.699892 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 31 14:48:10.700268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.700246 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-29w9c\"" Jul 31 14:48:10.700361 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.700268 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:10.700361 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.700333 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:10.700503 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.700485 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 31 14:48:10.700659 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.700620 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 31 14:48:10.702533 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.702507 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:10.705124 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.705103 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:10.705124 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.705116 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 31 14:48:10.705329 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.705181 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:10.705329 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.705191 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 31 14:48:10.705434 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.705336 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rrpbl\"" Jul 31 14:48:10.708173 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.708143 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.710433 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.710411 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:10.710569 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.710479 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:10.710704 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.710689 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 31 14:48:10.710942 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.710751 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6kr7w\"" Jul 31 14:48:10.721125 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.720913 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9wzwq" Jul 31 14:48:10.728952 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.728932 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9wzwq" Jul 31 14:48:10.775570 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.775510 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"489ee1b4953de705708ca3f9e7a9eb02","Type":"ContainerStarted","Data":"06237c5cc9e41c78aa4ddc718ba4baf4d101e8e595e4fcda3f7b789705c74413"} Jul 31 14:48:10.777006 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.776980 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" event={"ID":"a155768c4417dbe455678d25197f0bfd","Type":"ContainerStarted","Data":"7708c4e69c8eeacf0e73964e69a6d8ca8ec36705bec6efd18511e3fae1aee233"} Jul 31 14:48:10.786513 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.786494 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 31 14:48:10.798222 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798196 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-etc-selinux\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.798332 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798238 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-cni-bin\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.798332 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798295 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-cni-bin\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.798447 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-os-release\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.798509 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-slash\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.798509 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798475 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-socket-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.798509 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-kubelet-config\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:10.798696 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798531 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.798696 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798535 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-slash\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.798696 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798504 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-os-release\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.798696 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-modprobe-d\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.798696 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-sys\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.798696 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-host\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.798696 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-var-lib-kubelet\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.799032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798742 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.799032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798778 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-daemon-config\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.799032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-system-cni-dir\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.799032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798857 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-system-cni-dir\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.799032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.799032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798912 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-run-ovn\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.799032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e09cf890-da96-482f-b294-ac836147471a-etc-tuned\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.799032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.798977 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45c05bd2-6b63-4a74-86a8-f5a43da4c960-host\") pod \"node-ca-s7km4\" (UID: \"45c05bd2-6b63-4a74-86a8-f5a43da4c960\") " pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:10.799032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799006 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45c05bd2-6b63-4a74-86a8-f5a43da4c960-serviceca\") pod \"node-ca-s7km4\" (UID: \"45c05bd2-6b63-4a74-86a8-f5a43da4c960\") " pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:10.799032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799030 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-run-ovn\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-run-openvswitch\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799071 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-cni-netd\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799077 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-run-openvswitch\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799096 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3b6fa83-155e-4597-a00a-6940a61cac29-hosts-file\") pod \"node-resolver-tmrz2\" (UID: \"a3b6fa83-155e-4597-a00a-6940a61cac29\") " pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799123 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-cni-netd\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwkwd\" (UniqueName: \"kubernetes.io/projected/fecac6ec-445b-476b-8618-197295dbe47c-kube-api-access-cwkwd\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vml7w\" (UniqueName: \"kubernetes.io/projected/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-kube-api-access-vml7w\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799185 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a356e358-2937-4638-8ae1-cb46daa39c33-iptables-alerter-script\") pod \"iptables-alerter-zzp5r\" (UID: \"a356e358-2937-4638-8ae1-cb46daa39c33\") " pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799211 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-kubelet\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799236 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-sys-fs\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799315 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-kubelet\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799329 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-hostroot\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a356e358-2937-4638-8ae1-cb46daa39c33-host-slash\") pod \"iptables-alerter-zzp5r\" (UID: \"a356e358-2937-4638-8ae1-cb46daa39c33\") " pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-run-systemd\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-env-overrides\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799465 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wqp\" (UniqueName: \"kubernetes.io/projected/e09cf890-da96-482f-b294-ac836147471a-kube-api-access-g2wqp\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.799516 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799484 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-run-systemd\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799495 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799524 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e09cf890-da96-482f-b294-ac836147471a-tmp\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799498 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a356e358-2937-4638-8ae1-cb46daa39c33-host-slash\") pod \"iptables-alerter-zzp5r\" (UID: \"a356e358-2937-4638-8ae1-cb46daa39c33\") " pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-system-cni-dir\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799589 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-run-netns\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799591 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-var-lib-cni-multus\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799706 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799753 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-sysctl-conf\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.800293 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.799819 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.800907 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.800324 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a356e358-2937-4638-8ae1-cb46daa39c33-iptables-alerter-script\") pod \"iptables-alerter-zzp5r\" (UID: \"a356e358-2937-4638-8ae1-cb46daa39c33\") " pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:10.800907 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.800602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.800907 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.800770 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-env-overrides\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.801055 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.800959 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f56ffbbe-15a6-4eaa-93fa-2c815a6cf635-agent-certs\") pod \"konnectivity-agent-b2xz8\" (UID: \"f56ffbbe-15a6-4eaa-93fa-2c815a6cf635\") " pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:10.801055 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801007 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f56ffbbe-15a6-4eaa-93fa-2c815a6cf635-konnectivity-ca\") pod \"konnectivity-agent-b2xz8\" (UID: \"f56ffbbe-15a6-4eaa-93fa-2c815a6cf635\") " pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:10.801167 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801062 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-cnibin\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.801167 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801089 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-device-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.801167 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-log-socket\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.801167 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-sysctl-d\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.801379 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801196 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.801379 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvvs4\" (UniqueName: \"kubernetes.io/projected/45c05bd2-6b63-4a74-86a8-f5a43da4c960-kube-api-access-vvvs4\") pod \"node-ca-s7km4\" (UID: \"45c05bd2-6b63-4a74-86a8-f5a43da4c960\") " pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:10.801379 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-socket-dir-parent\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.801379 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-868gz\" (UniqueName: \"kubernetes.io/projected/b3394f34-2e60-4e63-aee0-84ed3f6dada9-kube-api-access-868gz\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:10.801379 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-systemd-units\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.801379 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-ovnkube-config\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.801779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-ovn-node-metrics-cert\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.801779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:10.801779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801507 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-cni-dir\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.801779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801549 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-var-lib-cni-bin\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.801779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-etc-kubernetes\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.801779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-cni-binary-copy\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.801779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s286j\" (UniqueName: \"kubernetes.io/projected/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-kube-api-access-s286j\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.801779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b6fa83-155e-4597-a00a-6940a61cac29-tmp-dir\") pod \"node-resolver-tmrz2\" (UID: \"a3b6fa83-155e-4597-a00a-6940a61cac29\") " pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:10.801779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801763 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6krv\" (UniqueName: \"kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv\") pod \"network-check-target-l4wr5\" (UID: \"4722e594-f22e-47e7-bd14-62ecccd174b1\") " pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801801 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6rh\" (UniqueName: \"kubernetes.io/projected/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-kube-api-access-fw6rh\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801835 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-systemd\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801871 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-os-release\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltd4\" (UniqueName: \"kubernetes.io/projected/a356e358-2937-4638-8ae1-cb46daa39c33-kube-api-access-vltd4\") pod \"iptables-alerter-zzp5r\" (UID: \"a356e358-2937-4638-8ae1-cb46daa39c33\") " pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-run-netns\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.801982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-etc-openvswitch\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802015 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-ovnkube-script-lib\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-kubernetes\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802089 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-run\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802125 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-cnibin\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-lib-modules\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802188 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhkk5\" (UniqueName: \"kubernetes.io/projected/a3b6fa83-155e-4597-a00a-6940a61cac29-kube-api-access-vhkk5\") pod \"node-resolver-tmrz2\" (UID: \"a3b6fa83-155e-4597-a00a-6940a61cac29\") " pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:10.802268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802255 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-conf-dir\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.802953 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802300 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-var-lib-openvswitch\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.802953 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802338 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-node-log\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.802953 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:10.802953 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-sysconfig\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.802953 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802445 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-var-lib-kubelet\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.802953 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802474 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-dbus\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:10.802953 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802510 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-cni-binary-copy\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.802953 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.802546 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-run-k8s-cni-cncf-io\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.803374 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.803118 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-run-multus-certs\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.803374 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.803152 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-registration-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.803490 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.803447 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-log-socket\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.803649 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.803617 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-node-log\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.803729 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.803708 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-systemd-units\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.804206 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.804187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-ovnkube-config\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.804323 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.804298 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-var-lib-openvswitch\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.804569 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.804524 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 31 14:48:10.804825 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.804803 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-cni-binary-copy\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.804825 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.804818 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-ovnkube-script-lib\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.805005 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.804876 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-etc-openvswitch\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.805099 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.805049 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-host-run-netns\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.805440 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.805420 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-cnibin\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.809353 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.809329 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-ovn-node-metrics-cert\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.812361 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.812318 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltd4\" (UniqueName: \"kubernetes.io/projected/a356e358-2937-4638-8ae1-cb46daa39c33-kube-api-access-vltd4\") pod \"iptables-alerter-zzp5r\" (UID: \"a356e358-2937-4638-8ae1-cb46daa39c33\") " pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:10.812730 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.812703 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6rh\" (UniqueName: \"kubernetes.io/projected/7b5438fc-3fae-4755-a7a2-f442f3b6ee42-kube-api-access-fw6rh\") pod \"ovnkube-node-2pdzx\" (UID: \"7b5438fc-3fae-4755-a7a2-f442f3b6ee42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:10.814051 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.814027 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s286j\" (UniqueName: \"kubernetes.io/projected/3d9ecebe-39b7-4f61-b0a3-aecbb90a9553-kube-api-access-s286j\") pod \"multus-additional-cni-plugins-84fbb\" (UID: \"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553\") " pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:10.903505 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-lib-modules\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.903645 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhkk5\" (UniqueName: \"kubernetes.io/projected/a3b6fa83-155e-4597-a00a-6940a61cac29-kube-api-access-vhkk5\") pod \"node-resolver-tmrz2\" (UID: \"a3b6fa83-155e-4597-a00a-6940a61cac29\") " pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:10.903645 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-conf-dir\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.903645 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903570 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:10.903645 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-sysconfig\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.903645 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903621 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-lib-modules\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.903645 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903635 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-var-lib-kubelet\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-dbus\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-cni-binary-copy\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-run-k8s-cni-cncf-io\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903747 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-run-multus-certs\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-registration-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903800 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-etc-selinux\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-socket-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903848 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-kubelet-config\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-modprobe-d\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903903 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-sys\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903929 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-host\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.903938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903933 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-dbus\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-var-lib-kubelet\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903989 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-daemon-config\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.904003 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904009 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-var-lib-kubelet\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e09cf890-da96-482f-b294-ac836147471a-etc-tuned\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904052 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-conf-dir\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.903902 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-sysconfig\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.904081 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs podName:b3394f34-2e60-4e63-aee0-84ed3f6dada9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:11.404046114 +0000 UTC m=+3.005482448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs") pod "network-metrics-daemon-thsf6" (UID: "b3394f34-2e60-4e63-aee0-84ed3f6dada9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904102 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-kubelet-config\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904119 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-host\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904170 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-run-multus-certs\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904216 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-registration-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904256 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-etc-selinux\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45c05bd2-6b63-4a74-86a8-f5a43da4c960-host\") pod \"node-ca-s7km4\" (UID: \"45c05bd2-6b63-4a74-86a8-f5a43da4c960\") " pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904286 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45c05bd2-6b63-4a74-86a8-f5a43da4c960-serviceca\") pod \"node-ca-s7km4\" (UID: \"45c05bd2-6b63-4a74-86a8-f5a43da4c960\") " pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904304 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3b6fa83-155e-4597-a00a-6940a61cac29-hosts-file\") pod \"node-resolver-tmrz2\" (UID: \"a3b6fa83-155e-4597-a00a-6940a61cac29\") " pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:10.904576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwkwd\" (UniqueName: \"kubernetes.io/projected/fecac6ec-445b-476b-8618-197295dbe47c-kube-api-access-cwkwd\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vml7w\" (UniqueName: \"kubernetes.io/projected/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-kube-api-access-vml7w\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-socket-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-sys-fs\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904396 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-hostroot\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wqp\" (UniqueName: \"kubernetes.io/projected/e09cf890-da96-482f-b294-ac836147471a-kube-api-access-g2wqp\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e09cf890-da96-482f-b294-ac836147471a-tmp\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904455 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-system-cni-dir\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904471 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-run-netns\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-var-lib-cni-multus\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-sysctl-conf\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f56ffbbe-15a6-4eaa-93fa-2c815a6cf635-agent-certs\") pod \"konnectivity-agent-b2xz8\" (UID: \"f56ffbbe-15a6-4eaa-93fa-2c815a6cf635\") " pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f56ffbbe-15a6-4eaa-93fa-2c815a6cf635-konnectivity-ca\") pod \"konnectivity-agent-b2xz8\" (UID: \"f56ffbbe-15a6-4eaa-93fa-2c815a6cf635\") " pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-cnibin\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-device-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904599 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-sysctl-d\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904705 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-cni-binary-copy\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.905573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904717 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904728 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45c05bd2-6b63-4a74-86a8-f5a43da4c960-serviceca\") pod \"node-ca-s7km4\" (UID: \"45c05bd2-6b63-4a74-86a8-f5a43da4c960\") " pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904759 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3b6fa83-155e-4597-a00a-6940a61cac29-hosts-file\") pod \"node-resolver-tmrz2\" (UID: \"a3b6fa83-155e-4597-a00a-6940a61cac29\") " pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904782 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-modprobe-d\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904793 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45c05bd2-6b63-4a74-86a8-f5a43da4c960-host\") pod \"node-ca-s7km4\" (UID: \"45c05bd2-6b63-4a74-86a8-f5a43da4c960\") " pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-run-netns\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904813 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-var-lib-cni-multus\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904817 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvvs4\" (UniqueName: \"kubernetes.io/projected/45c05bd2-6b63-4a74-86a8-f5a43da4c960-kube-api-access-vvvs4\") pod \"node-ca-s7km4\" (UID: \"45c05bd2-6b63-4a74-86a8-f5a43da4c960\") " pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-socket-dir-parent\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-cnibin\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904896 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-868gz\" (UniqueName: \"kubernetes.io/projected/b3394f34-2e60-4e63-aee0-84ed3f6dada9-kube-api-access-868gz\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904929 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904941 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-device-dir\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-cni-dir\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.904986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-var-lib-cni-bin\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905037 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-sys\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905046 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-sysctl-d\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905097 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fecac6ec-445b-476b-8618-197295dbe47c-sys-fs\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.906561 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905141 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-hostroot\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-daemon-config\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-etc-kubernetes\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b6fa83-155e-4597-a00a-6940a61cac29-tmp-dir\") pod \"node-resolver-tmrz2\" (UID: \"a3b6fa83-155e-4597-a00a-6940a61cac29\") " pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6krv\" (UniqueName: \"kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv\") pod \"network-check-target-l4wr5\" (UID: \"4722e594-f22e-47e7-bd14-62ecccd174b1\") " pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-systemd\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905602 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-os-release\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-kubernetes\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905666 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-socket-dir-parent\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905039 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-var-lib-cni-bin\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905553 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f56ffbbe-15a6-4eaa-93fa-2c815a6cf635-konnectivity-ca\") pod \"konnectivity-agent-b2xz8\" (UID: \"f56ffbbe-15a6-4eaa-93fa-2c815a6cf635\") " pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-run\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.905691 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.905777 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret podName:fa25e8f1-1145-4595-ae2a-5733ae0fc8fb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:11.40575326 +0000 UTC m=+3.007189577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret") pod "global-pull-secret-syncer-tx9n9" (UID: "fa25e8f1-1145-4595-ae2a-5733ae0fc8fb") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905369 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-sysctl-conf\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905832 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-system-cni-dir\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905866 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-os-release\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905877 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-var-lib-kubelet\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.907443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905404 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-host-run-k8s-cni-cncf-io\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.908395 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.905437 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-etc-kubernetes\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.908395 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.906004 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-run\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.908395 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.906153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-kubernetes\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.908395 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.906174 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b6fa83-155e-4597-a00a-6940a61cac29-tmp-dir\") pod \"node-resolver-tmrz2\" (UID: \"a3b6fa83-155e-4597-a00a-6940a61cac29\") " pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:10.908395 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.906209 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-multus-cni-dir\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.908395 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.906268 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e09cf890-da96-482f-b294-ac836147471a-etc-systemd\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.908395 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.906569 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e09cf890-da96-482f-b294-ac836147471a-etc-tuned\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.908395 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.908378 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e09cf890-da96-482f-b294-ac836147471a-tmp\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.908818 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.908719 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f56ffbbe-15a6-4eaa-93fa-2c815a6cf635-agent-certs\") pod \"konnectivity-agent-b2xz8\" (UID: \"f56ffbbe-15a6-4eaa-93fa-2c815a6cf635\") " pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:10.911939 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.911912 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:48:10.912052 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.911944 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:48:10.912052 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.911958 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z6krv for pod openshift-network-diagnostics/network-check-target-l4wr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:10.912052 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:10.912031 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv podName:4722e594-f22e-47e7-bd14-62ecccd174b1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:11.41201394 +0000 UTC m=+3.013450277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z6krv" (UniqueName: "kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv") pod "network-check-target-l4wr5" (UID: "4722e594-f22e-47e7-bd14-62ecccd174b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:10.913395 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.913371 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhkk5\" (UniqueName: \"kubernetes.io/projected/a3b6fa83-155e-4597-a00a-6940a61cac29-kube-api-access-vhkk5\") pod \"node-resolver-tmrz2\" (UID: \"a3b6fa83-155e-4597-a00a-6940a61cac29\") " pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:10.914180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.914136 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vml7w\" (UniqueName: \"kubernetes.io/projected/0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28-kube-api-access-vml7w\") pod \"multus-jjklb\" (UID: \"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28\") " pod="openshift-multus/multus-jjklb" Jul 31 14:48:10.915322 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.915297 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-868gz\" (UniqueName: \"kubernetes.io/projected/b3394f34-2e60-4e63-aee0-84ed3f6dada9-kube-api-access-868gz\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:10.915967 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.915944 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwkwd\" (UniqueName: \"kubernetes.io/projected/fecac6ec-445b-476b-8618-197295dbe47c-kube-api-access-cwkwd\") pod \"aws-ebs-csi-driver-node-p687s\" (UID: \"fecac6ec-445b-476b-8618-197295dbe47c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:10.916152 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.916131 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvvs4\" (UniqueName: \"kubernetes.io/projected/45c05bd2-6b63-4a74-86a8-f5a43da4c960-kube-api-access-vvvs4\") pod \"node-ca-s7km4\" (UID: \"45c05bd2-6b63-4a74-86a8-f5a43da4c960\") " pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:10.917187 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.917163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wqp\" (UniqueName: \"kubernetes.io/projected/e09cf890-da96-482f-b294-ac836147471a-kube-api-access-g2wqp\") pod \"tuned-x9vd6\" (UID: \"e09cf890-da96-482f-b294-ac836147471a\") " pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:10.996951 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:10.996861 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84fbb" Jul 31 14:48:11.009068 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:11.009020 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9ecebe_39b7_4f61_b0a3_aecbb90a9553.slice/crio-fdafb85e7b1ad5ca79e85258bda0bf1230bc6ee10b30ae1435d5e0ba4647f55c WatchSource:0}: Error finding container fdafb85e7b1ad5ca79e85258bda0bf1230bc6ee10b30ae1435d5e0ba4647f55c: Status 404 returned error can't find the container with id fdafb85e7b1ad5ca79e85258bda0bf1230bc6ee10b30ae1435d5e0ba4647f55c Jul 31 14:48:11.009068 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.009046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zzp5r" Jul 31 14:48:11.016916 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.016891 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:11.018849 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:11.018823 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda356e358_2937_4638_8ae1_cb46daa39c33.slice/crio-d4abc3939071b9bcb2eb8e165292f9d7e2ac396a4c3e378e3a06e8b0215e7fb5 WatchSource:0}: Error finding container d4abc3939071b9bcb2eb8e165292f9d7e2ac396a4c3e378e3a06e8b0215e7fb5: Status 404 returned error can't find the container with id d4abc3939071b9bcb2eb8e165292f9d7e2ac396a4c3e378e3a06e8b0215e7fb5 Jul 31 14:48:11.024695 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.024624 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" Jul 31 14:48:11.026682 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:11.026660 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5438fc_3fae_4755_a7a2_f442f3b6ee42.slice/crio-5c357b6d549a70993fb2a189e98fbccee79800a5fbf3741f7673b2d5dba133f0 WatchSource:0}: Error finding container 5c357b6d549a70993fb2a189e98fbccee79800a5fbf3741f7673b2d5dba133f0: Status 404 returned error can't find the container with id 5c357b6d549a70993fb2a189e98fbccee79800a5fbf3741f7673b2d5dba133f0 Jul 31 14:48:11.031993 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.031970 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tmrz2" Jul 31 14:48:11.036272 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:11.036248 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09cf890_da96_482f_b294_ac836147471a.slice/crio-ca1f4cf59ab92ef3055b9e5c0f84553e43cc6d8d24d7c4b97022f69c70e81339 WatchSource:0}: Error finding container ca1f4cf59ab92ef3055b9e5c0f84553e43cc6d8d24d7c4b97022f69c70e81339: Status 404 returned error can't find the container with id ca1f4cf59ab92ef3055b9e5c0f84553e43cc6d8d24d7c4b97022f69c70e81339 Jul 31 14:48:11.041217 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.041198 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s7km4" Jul 31 14:48:11.043146 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:11.043123 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b6fa83_155e_4597_a00a_6940a61cac29.slice/crio-f7d25dc65762591a1af103d357094dc05763dea3d7af4123f660c65cc1a6c66c WatchSource:0}: Error finding container f7d25dc65762591a1af103d357094dc05763dea3d7af4123f660c65cc1a6c66c: Status 404 returned error can't find the container with id f7d25dc65762591a1af103d357094dc05763dea3d7af4123f660c65cc1a6c66c Jul 31 14:48:11.049980 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.049887 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" Jul 31 14:48:11.052619 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:11.052592 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c05bd2_6b63_4a74_86a8_f5a43da4c960.slice/crio-cc97360dc5e7342443f3f3527b9e480e8e09de7ae3f5367e5570b63b165ee5df WatchSource:0}: Error finding container cc97360dc5e7342443f3f3527b9e480e8e09de7ae3f5367e5570b63b165ee5df: Status 404 returned error can't find the container with id cc97360dc5e7342443f3f3527b9e480e8e09de7ae3f5367e5570b63b165ee5df Jul 31 14:48:11.058735 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.058717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:11.059555 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:11.059531 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecac6ec_445b_476b_8618_197295dbe47c.slice/crio-ff58eacbb9a2d7231e45feaa918fcacc5bb3a8b6d9661a9caf3a63981db9c4e8 WatchSource:0}: Error finding container ff58eacbb9a2d7231e45feaa918fcacc5bb3a8b6d9661a9caf3a63981db9c4e8: Status 404 returned error can't find the container with id ff58eacbb9a2d7231e45feaa918fcacc5bb3a8b6d9661a9caf3a63981db9c4e8 Jul 31 14:48:11.065665 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.065631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jjklb" Jul 31 14:48:11.070647 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:11.070606 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56ffbbe_15a6_4eaa_93fa_2c815a6cf635.slice/crio-b4b2ac186eebee4cd6d6f9a01332bebacfe29ce64b24819086d327c5e12bab92 WatchSource:0}: Error finding container b4b2ac186eebee4cd6d6f9a01332bebacfe29ce64b24819086d327c5e12bab92: Status 404 returned error can't find the container with id b4b2ac186eebee4cd6d6f9a01332bebacfe29ce64b24819086d327c5e12bab92 Jul 31 14:48:11.076147 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:11.076127 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0c9dc9_912a_4ba8_9ed1_acfc78a03d28.slice/crio-fc898f985414a2d364a1f4ef256b6332b2f80d10ec07e5e610a76346ff1e5786 WatchSource:0}: Error finding container fc898f985414a2d364a1f4ef256b6332b2f80d10ec07e5e610a76346ff1e5786: Status 404 returned error can't find the container with id fc898f985414a2d364a1f4ef256b6332b2f80d10ec07e5e610a76346ff1e5786 Jul 31 14:48:11.409226 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.409099 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:11.409226 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.409169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:11.409457 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:11.409263 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:11.409457 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:11.409267 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:11.409457 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:11.409326 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs podName:b3394f34-2e60-4e63-aee0-84ed3f6dada9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:12.409306611 +0000 UTC m=+4.010742930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs") pod "network-metrics-daemon-thsf6" (UID: "b3394f34-2e60-4e63-aee0-84ed3f6dada9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:11.409457 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:11.409349 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret podName:fa25e8f1-1145-4595-ae2a-5733ae0fc8fb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:12.409341432 +0000 UTC m=+4.010777748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret") pod "global-pull-secret-syncer-tx9n9" (UID: "fa25e8f1-1145-4595-ae2a-5733ae0fc8fb") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:11.510242 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.510210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6krv\" (UniqueName: \"kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv\") pod \"network-check-target-l4wr5\" (UID: \"4722e594-f22e-47e7-bd14-62ecccd174b1\") " pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:11.510446 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:11.510428 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:48:11.510521 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:11.510457 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:48:11.510521 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:11.510471 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z6krv for pod openshift-network-diagnostics/network-check-target-l4wr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:11.510658 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:11.510533 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv podName:4722e594-f22e-47e7-bd14-62ecccd174b1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:12.510513349 +0000 UTC m=+4.111949681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6krv" (UniqueName: "kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv") pod "network-check-target-l4wr5" (UID: "4722e594-f22e-47e7-bd14-62ecccd174b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:11.732583 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.732392 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:10 +0000 UTC" deadline="2028-02-08 02:58:51.295385465 +0000 UTC" Jul 31 14:48:11.732583 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.732461 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13356h10m39.562929829s" Jul 31 14:48:11.770350 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.769784 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:11.770350 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:11.769920 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:11.780863 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.780786 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" event={"ID":"fecac6ec-445b-476b-8618-197295dbe47c","Type":"ContainerStarted","Data":"ff58eacbb9a2d7231e45feaa918fcacc5bb3a8b6d9661a9caf3a63981db9c4e8"} Jul 31 14:48:11.782713 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.782622 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tmrz2" event={"ID":"a3b6fa83-155e-4597-a00a-6940a61cac29","Type":"ContainerStarted","Data":"f7d25dc65762591a1af103d357094dc05763dea3d7af4123f660c65cc1a6c66c"} Jul 31 14:48:11.785026 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.784525 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" event={"ID":"7b5438fc-3fae-4755-a7a2-f442f3b6ee42","Type":"ContainerStarted","Data":"5c357b6d549a70993fb2a189e98fbccee79800a5fbf3741f7673b2d5dba133f0"} Jul 31 14:48:11.786512 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.786197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fbb" event={"ID":"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553","Type":"ContainerStarted","Data":"fdafb85e7b1ad5ca79e85258bda0bf1230bc6ee10b30ae1435d5e0ba4647f55c"} Jul 31 14:48:11.788240 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.787769 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jjklb" event={"ID":"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28","Type":"ContainerStarted","Data":"fc898f985414a2d364a1f4ef256b6332b2f80d10ec07e5e610a76346ff1e5786"} Jul 31 14:48:11.792844 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.790266 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b2xz8" event={"ID":"f56ffbbe-15a6-4eaa-93fa-2c815a6cf635","Type":"ContainerStarted","Data":"b4b2ac186eebee4cd6d6f9a01332bebacfe29ce64b24819086d327c5e12bab92"} Jul 31 14:48:11.797330 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.797302 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s7km4" event={"ID":"45c05bd2-6b63-4a74-86a8-f5a43da4c960","Type":"ContainerStarted","Data":"cc97360dc5e7342443f3f3527b9e480e8e09de7ae3f5367e5570b63b165ee5df"} Jul 31 14:48:11.798776 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.798752 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" event={"ID":"e09cf890-da96-482f-b294-ac836147471a","Type":"ContainerStarted","Data":"ca1f4cf59ab92ef3055b9e5c0f84553e43cc6d8d24d7c4b97022f69c70e81339"} Jul 31 14:48:11.800579 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:11.800554 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zzp5r" event={"ID":"a356e358-2937-4638-8ae1-cb46daa39c33","Type":"ContainerStarted","Data":"d4abc3939071b9bcb2eb8e165292f9d7e2ac396a4c3e378e3a06e8b0215e7fb5"} Jul 31 14:48:12.418576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:12.418537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:12.418794 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:12.418627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:12.418794 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:12.418750 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:12.418915 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:12.418813 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs podName:b3394f34-2e60-4e63-aee0-84ed3f6dada9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:14.418792639 +0000 UTC m=+6.020228962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs") pod "network-metrics-daemon-thsf6" (UID: "b3394f34-2e60-4e63-aee0-84ed3f6dada9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:12.418915 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:12.418852 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:12.419028 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:12.418919 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret podName:fa25e8f1-1145-4595-ae2a-5733ae0fc8fb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:14.418901044 +0000 UTC m=+6.020337378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret") pod "global-pull-secret-syncer-tx9n9" (UID: "fa25e8f1-1145-4595-ae2a-5733ae0fc8fb") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:12.519918 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:12.519884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6krv\" (UniqueName: \"kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv\") pod \"network-check-target-l4wr5\" (UID: \"4722e594-f22e-47e7-bd14-62ecccd174b1\") " pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:12.520086 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:12.520060 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:48:12.520086 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:12.520080 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:48:12.520211 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:12.520094 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z6krv for pod openshift-network-diagnostics/network-check-target-l4wr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:12.520211 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:12.520153 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv podName:4722e594-f22e-47e7-bd14-62ecccd174b1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:14.520135516 +0000 UTC m=+6.121571839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6krv" (UniqueName: "kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv") pod "network-check-target-l4wr5" (UID: "4722e594-f22e-47e7-bd14-62ecccd174b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:12.733085 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:12.732995 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:10 +0000 UTC" deadline="2028-01-22 00:27:31.271921451 +0000 UTC" Jul 31 14:48:12.733085 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:12.733035 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12945h39m18.538890615s" Jul 31 14:48:12.772816 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:12.772132 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:12.772816 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:12.772276 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:12.773395 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:12.773122 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:12.773395 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:12.773233 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:12.815138 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:12.815105 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:48:13.770004 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:13.769975 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:13.770468 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:13.770098 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:14.442410 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:14.441643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:14.442410 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:14.441722 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:14.442410 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:14.441860 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:14.442410 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:14.441924 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret podName:fa25e8f1-1145-4595-ae2a-5733ae0fc8fb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:18.441903804 +0000 UTC m=+10.043340132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret") pod "global-pull-secret-syncer-tx9n9" (UID: "fa25e8f1-1145-4595-ae2a-5733ae0fc8fb") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:14.442410 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:14.442326 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:14.442410 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:14.442376 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs podName:b3394f34-2e60-4e63-aee0-84ed3f6dada9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:18.442360224 +0000 UTC m=+10.043796545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs") pod "network-metrics-daemon-thsf6" (UID: "b3394f34-2e60-4e63-aee0-84ed3f6dada9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:14.543080 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:14.542491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6krv\" (UniqueName: \"kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv\") pod \"network-check-target-l4wr5\" (UID: \"4722e594-f22e-47e7-bd14-62ecccd174b1\") " pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:14.543080 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:14.542674 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:48:14.543080 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:14.542697 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:48:14.543080 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:14.542709 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z6krv for pod openshift-network-diagnostics/network-check-target-l4wr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:14.543080 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:14.542762 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv podName:4722e594-f22e-47e7-bd14-62ecccd174b1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:18.542744889 +0000 UTC m=+10.144181226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6krv" (UniqueName: "kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv") pod "network-check-target-l4wr5" (UID: "4722e594-f22e-47e7-bd14-62ecccd174b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:14.771310 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:14.770644 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:14.771310 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:14.770775 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:14.771965 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:14.771893 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:14.772065 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:14.772025 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:15.769853 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:15.769819 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:15.770016 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:15.769953 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:16.770177 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:16.769725 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:16.770177 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:16.769850 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:16.770784 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:16.770756 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:16.770896 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:16.770873 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:17.770079 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:17.770028 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:17.770242 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:17.770157 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:18.473325 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:18.473283 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:18.473500 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:18.473364 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:18.473500 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:18.473495 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:18.473620 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:18.473574 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret podName:fa25e8f1-1145-4595-ae2a-5733ae0fc8fb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.473540549 +0000 UTC m=+18.074976887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret") pod "global-pull-secret-syncer-tx9n9" (UID: "fa25e8f1-1145-4595-ae2a-5733ae0fc8fb") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:18.473811 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:18.473787 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:18.473891 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:18.473856 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs podName:b3394f34-2e60-4e63-aee0-84ed3f6dada9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.473839324 +0000 UTC m=+18.075275651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs") pod "network-metrics-daemon-thsf6" (UID: "b3394f34-2e60-4e63-aee0-84ed3f6dada9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:18.573972 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:18.573881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6krv\" (UniqueName: \"kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv\") pod \"network-check-target-l4wr5\" (UID: \"4722e594-f22e-47e7-bd14-62ecccd174b1\") " pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:18.574125 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:18.574074 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:48:18.574125 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:18.574097 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:48:18.574125 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:18.574107 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z6krv for pod openshift-network-diagnostics/network-check-target-l4wr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:18.574293 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:18.574148 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv podName:4722e594-f22e-47e7-bd14-62ecccd174b1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.574135747 +0000 UTC m=+18.175572063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6krv" (UniqueName: "kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv") pod "network-check-target-l4wr5" (UID: "4722e594-f22e-47e7-bd14-62ecccd174b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:18.769623 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:18.769374 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:18.769623 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:18.769533 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:18.771116 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:18.770970 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:18.771116 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:18.771081 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:19.769210 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:19.769166 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:19.769388 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:19.769295 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:20.770260 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:20.770223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:20.770260 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:20.770263 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:20.770774 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:20.770368 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:20.770774 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:20.770481 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:21.769595 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:21.769567 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:21.769729 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:21.769656 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:22.769643 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:22.769603 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:22.770101 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:22.769608 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:22.770101 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:22.769721 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:22.770101 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:22.769787 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:23.770039 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:23.770008 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:23.770416 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:23.770126 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:24.769973 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:24.769949 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:24.770150 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:24.769946 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:24.770150 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:24.770077 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:24.770150 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:24.770120 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:25.769874 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:25.769841 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:25.770053 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:25.769931 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:26.535339 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:26.535307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:26.535862 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:26.535377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:26.535862 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:26.535417 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:26.535862 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:26.535483 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:26.535862 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:26.535485 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret podName:fa25e8f1-1145-4595-ae2a-5733ae0fc8fb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:42.535464938 +0000 UTC m=+34.136901254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret") pod "global-pull-secret-syncer-tx9n9" (UID: "fa25e8f1-1145-4595-ae2a-5733ae0fc8fb") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:26.535862 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:26.535545 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs podName:b3394f34-2e60-4e63-aee0-84ed3f6dada9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:42.535529013 +0000 UTC m=+34.136965342 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs") pod "network-metrics-daemon-thsf6" (UID: "b3394f34-2e60-4e63-aee0-84ed3f6dada9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:26.635686 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:26.635653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6krv\" (UniqueName: \"kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv\") pod \"network-check-target-l4wr5\" (UID: \"4722e594-f22e-47e7-bd14-62ecccd174b1\") " pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:26.635851 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:26.635826 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:48:26.635851 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:26.635846 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:48:26.635958 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:26.635859 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z6krv for pod openshift-network-diagnostics/network-check-target-l4wr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:26.635958 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:26.635920 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv podName:4722e594-f22e-47e7-bd14-62ecccd174b1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:42.635901689 +0000 UTC m=+34.237338020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6krv" (UniqueName: "kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv") pod "network-check-target-l4wr5" (UID: "4722e594-f22e-47e7-bd14-62ecccd174b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:26.769227 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:26.769200 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:26.769410 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:26.769233 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:26.769410 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:26.769320 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:26.769532 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:26.769409 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:27.769401 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:27.769373 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:27.769814 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:27.769470 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:28.770934 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:28.770900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:28.771351 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:28.771012 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:28.771351 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:28.771079 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:28.771351 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:28.771174 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:29.769331 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:29.769301 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:29.769497 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:29.769387 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:30.770243 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.769955 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:30.770799 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.770039 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:30.770799 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:30.770318 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:30.770799 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:30.770442 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:30.839200 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.839171 2582 generic.go:356] "Generic (PLEG): container finished" podID="3d9ecebe-39b7-4f61-b0a3-aecbb90a9553" containerID="0f25aecaa640b6d71a15b25f9d6543ab9ed017227fd013509a5bf99a46ee42c0" exitCode=0 Jul 31 14:48:30.839328 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.839257 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fbb" event={"ID":"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553","Type":"ContainerDied","Data":"0f25aecaa640b6d71a15b25f9d6543ab9ed017227fd013509a5bf99a46ee42c0"} Jul 31 14:48:30.841110 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.841087 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jjklb" event={"ID":"0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28","Type":"ContainerStarted","Data":"4798305f78fe4c945cd0c7c8b776d1a23d890a6bfa3f0e90d078a6db0444f5c6"} Jul 31 14:48:30.842536 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.842518 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b2xz8" event={"ID":"f56ffbbe-15a6-4eaa-93fa-2c815a6cf635","Type":"ContainerStarted","Data":"e75ea0161a3cecdcef21c921033506dac289f38b196d0f33974ba6008ca4cd89"} Jul 31 14:48:30.843931 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.843904 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s7km4" event={"ID":"45c05bd2-6b63-4a74-86a8-f5a43da4c960","Type":"ContainerStarted","Data":"afc9a840c3c5c4387d37b8e09e6b9ed6a2e01c417b687a68f864116e04c8782d"} Jul 31 14:48:30.845404 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.845362 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" event={"ID":"e09cf890-da96-482f-b294-ac836147471a","Type":"ContainerStarted","Data":"0f051aa563b0dabd26802a693f17e029712f3762d0a21e4828b721c89dcfa3c4"} Jul 31 14:48:30.847066 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.847039 2582 generic.go:356] "Generic (PLEG): container finished" podID="489ee1b4953de705708ca3f9e7a9eb02" containerID="6ef8672427889681ff0b7cdec906e0903c9cfc1d8cf3c1a92cbdb95ec2766d35" exitCode=0 Jul 31 14:48:30.847174 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.847111 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"489ee1b4953de705708ca3f9e7a9eb02","Type":"ContainerDied","Data":"6ef8672427889681ff0b7cdec906e0903c9cfc1d8cf3c1a92cbdb95ec2766d35"} Jul 31 14:48:30.848493 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.848475 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" event={"ID":"a155768c4417dbe455678d25197f0bfd","Type":"ContainerStarted","Data":"b7f301ccc623cec9cdcf9ed25c48b3e1fbba4f173297f3dde9fa3c27129b508a"} Jul 31 14:48:30.850133 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.850111 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" event={"ID":"fecac6ec-445b-476b-8618-197295dbe47c","Type":"ContainerStarted","Data":"30752cb3c604281b16154e7e72750ee5aadc3af0140bf6b8aa0b18e732e88a38"} Jul 31 14:48:30.851590 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.851547 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tmrz2" event={"ID":"a3b6fa83-155e-4597-a00a-6940a61cac29","Type":"ContainerStarted","Data":"e9f302951667e4bf9d3013d510091b9deed694037cb898970a378cbb0d04749d"} Jul 31 14:48:30.854350 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.854331 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" event={"ID":"7b5438fc-3fae-4755-a7a2-f442f3b6ee42","Type":"ContainerStarted","Data":"1e65e8ed521c1d093c2c0860bc3fbdb18eff28474a8f7fe10b4a4d8fbe5b747b"} Jul 31 14:48:30.854442 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.854388 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" event={"ID":"7b5438fc-3fae-4755-a7a2-f442f3b6ee42","Type":"ContainerStarted","Data":"d78389a2b8a4b92ff2271b6e50b74956b7a7e6a6b8f9d22b70fdc393e9a0b834"} Jul 31 14:48:30.854442 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.854417 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" event={"ID":"7b5438fc-3fae-4755-a7a2-f442f3b6ee42","Type":"ContainerStarted","Data":"65aaf413fea51549e23bce6749f36f248940ce370445c622c8a183d437eb8492"} Jul 31 14:48:30.854523 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.854447 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" event={"ID":"7b5438fc-3fae-4755-a7a2-f442f3b6ee42","Type":"ContainerStarted","Data":"d8764d31c97e974ea0b7594bce2e74f567f7da28bee4509999f04f8499e9667e"} Jul 31 14:48:30.854523 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.854474 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" event={"ID":"7b5438fc-3fae-4755-a7a2-f442f3b6ee42","Type":"ContainerStarted","Data":"859050f8e1339cd09c4a4814e0b805e9f9b62ba5467c262552c22e8052e12541"} Jul 31 14:48:30.854523 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.854502 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" event={"ID":"7b5438fc-3fae-4755-a7a2-f442f3b6ee42","Type":"ContainerStarted","Data":"12a619ad8f85e698391f10dc560860932125756467ff61f6b83819ac5684a99b"} Jul 31 14:48:30.923569 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.923503 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" podStartSLOduration=20.92348636 podStartE2EDuration="20.92348636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:30.895658505 +0000 UTC m=+22.497094842" watchObservedRunningTime="2026-07-31 14:48:30.92348636 +0000 UTC m=+22.524922698" Jul 31 14:48:30.940803 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.940771 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x9vd6" podStartSLOduration=2.922385518 podStartE2EDuration="21.94075933s" totalImagesPullingTime="19.018373812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:30.922596398 +0000 UTC m=+22.524032736" watchObservedRunningTime="2026-07-31 14:48:30.94075933 +0000 UTC m=+22.542195666" Jul 31 14:48:30.969041 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.969006 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s7km4" podStartSLOduration=2.969965279 podStartE2EDuration="21.968992856s" totalImagesPullingTime="18.999027577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:30.941601263 +0000 UTC m=+22.543037600" watchObservedRunningTime="2026-07-31 14:48:30.968992856 +0000 UTC m=+22.570429197" Jul 31 14:48:30.969478 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:30.969303 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jjklb" podStartSLOduration=2.863487971 podStartE2EDuration="21.969291021s" totalImagesPullingTime="19.10580305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:30.968792741 +0000 UTC m=+22.570229078" watchObservedRunningTime="2026-07-31 14:48:30.969291021 +0000 UTC m=+22.570727359" Jul 31 14:48:31.020174 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:31.020127 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tmrz2" podStartSLOduration=3.011006445 podStartE2EDuration="22.020106694s" totalImagesPullingTime="19.009100249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:31.019456905 +0000 UTC m=+22.620893243" watchObservedRunningTime="2026-07-31 14:48:31.020106694 +0000 UTC m=+22.621543034" Jul 31 14:48:31.047330 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:31.047254 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b2xz8" podStartSLOduration=11.177409922 podStartE2EDuration="22.047235651s" totalImagesPullingTime="10.869825729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:31.047038424 +0000 UTC m=+22.648474761" watchObservedRunningTime="2026-07-31 14:48:31.047235651 +0000 UTC m=+22.648671989" Jul 31 14:48:31.769859 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:31.769840 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:31.769964 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:31.769941 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:31.858577 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:31.858497 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zzp5r" event={"ID":"a356e358-2937-4638-8ae1-cb46daa39c33","Type":"ContainerStarted","Data":"53adda59534b8d1053474468f336f4cbd08bb5c8b42b4e6f2fe86ac33158c20c"} Jul 31 14:48:31.860749 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:31.860657 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"489ee1b4953de705708ca3f9e7a9eb02","Type":"ContainerStarted","Data":"17ab9c0caa3087281012ac767f9a8178e1e52e3f3ff6de2ce8e00a936d7e0f14"} Jul 31 14:48:31.876346 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:31.876296 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zzp5r" podStartSLOduration=3.84159325 podStartE2EDuration="22.876274623s" totalImagesPullingTime="19.034681373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:31.875931995 +0000 UTC m=+23.477368332" watchObservedRunningTime="2026-07-31 14:48:31.876274623 +0000 UTC m=+23.477710961" Jul 31 14:48:31.895875 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:31.895832 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" podStartSLOduration=21.895815994 podStartE2EDuration="21.895815994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:31.895193041 +0000 UTC m=+23.496629378" watchObservedRunningTime="2026-07-31 14:48:31.895815994 +0000 UTC m=+23.497252332" Jul 31 14:48:31.925808 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:31.925783 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 31 14:48:32.753945 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:32.753780 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T14:48:31.925800866Z","UUID":"7746d184-d7fb-42e9-9994-a5495025e1ec","Handler":null,"Name":"","Endpoint":""} Jul 31 14:48:32.755577 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:32.755523 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 31 14:48:32.755577 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:32.755580 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 31 14:48:32.769441 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:32.769407 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:32.769593 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:32.769411 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:32.769593 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:32.769526 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:32.769753 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:32.769637 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:32.864734 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:32.864705 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" event={"ID":"fecac6ec-445b-476b-8618-197295dbe47c","Type":"ContainerStarted","Data":"e591dcc4d7bf6a10ed1a8fa462fb13f8343284f1f07a3602d186f0342dfff8f6"} Jul 31 14:48:33.599603 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.599323 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:33.600062 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.600042 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:33.700336 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.700298 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:33.770131 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.770106 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:33.770289 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:33.770213 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:33.800932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.800907 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:33.869006 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.868930 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" event={"ID":"fecac6ec-445b-476b-8618-197295dbe47c","Type":"ContainerStarted","Data":"7db79a27f4709006d657e3a0f00a8cd8f3f50ed13dbb3b7682bb144d9846204f"} Jul 31 14:48:33.873076 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.872966 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" event={"ID":"7b5438fc-3fae-4755-a7a2-f442f3b6ee42","Type":"ContainerStarted","Data":"0448573cef1198e6a53e6c91183eb8c5a5872c0f98555be654d74d1c9d96e890"} Jul 31 14:48:33.873349 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.873111 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:33.873776 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.873759 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b2xz8" Jul 31 14:48:33.887447 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.887405 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p687s" podStartSLOduration=3.391707129 podStartE2EDuration="24.887391874s" totalImagesPullingTime="21.495684745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:33.887345548 +0000 UTC m=+25.488781886" watchObservedRunningTime="2026-07-31 14:48:33.887391874 +0000 UTC m=+25.488828210" Jul 31 14:48:33.963793 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:33.963764 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:34.770153 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:34.770113 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:34.770318 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:34.770248 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:34.770381 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:34.770333 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:34.770444 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:34.770426 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:35.770286 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:35.770117 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:35.770920 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:35.770351 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:35.882413 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:35.882377 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" event={"ID":"7b5438fc-3fae-4755-a7a2-f442f3b6ee42","Type":"ContainerStarted","Data":"94a06b0eeef7ac6a25afdb631fef08868dfd58bfbe99f60681a2dcfefc804bb0"} Jul 31 14:48:35.882842 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:35.882590 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:35.884253 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:35.884218 2582 generic.go:356] "Generic (PLEG): container finished" podID="3d9ecebe-39b7-4f61-b0a3-aecbb90a9553" containerID="4154a46be901672a6342ede42df2a51864c50b5dfe4f705b1bda5ff3ef9cd00e" exitCode=0 Jul 31 14:48:35.884345 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:35.884253 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fbb" event={"ID":"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553","Type":"ContainerDied","Data":"4154a46be901672a6342ede42df2a51864c50b5dfe4f705b1bda5ff3ef9cd00e"} Jul 31 14:48:35.897521 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:35.897494 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:35.898861 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:35.898842 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:35.913746 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:35.913702 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" podStartSLOduration=7.842402218 podStartE2EDuration="26.913691289s" totalImagesPullingTime="19.071289071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:35.913239853 +0000 UTC m=+27.514676202" watchObservedRunningTime="2026-07-31 14:48:35.913691289 +0000 UTC m=+27.515127626" Jul 31 14:48:35.954871 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:35.954849 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:36.769656 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:36.769539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:36.769947 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:36.769917 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:36.770075 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:36.769962 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:36.770135 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:36.770082 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:36.886649 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:36.886624 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:36.887057 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:36.886691 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:36.901982 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:36.901900 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:48:37.177657 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:37.177467 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-thsf6"] Jul 31 14:48:37.177747 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:37.177735 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:37.177844 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:37.177823 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:37.181553 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:37.181534 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l4wr5"] Jul 31 14:48:37.181663 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:37.181631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:37.181708 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:37.181692 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:37.189709 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:37.189688 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tx9n9"] Jul 31 14:48:37.189803 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:37.189781 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:37.189900 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:37.189874 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:37.890285 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:37.890250 2582 generic.go:356] "Generic (PLEG): container finished" podID="3d9ecebe-39b7-4f61-b0a3-aecbb90a9553" containerID="b0233b77826167f044d3183010b2b2c843853795f2e3a781543254b20a68fa3e" exitCode=0 Jul 31 14:48:37.890642 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:37.890317 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fbb" event={"ID":"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553","Type":"ContainerDied","Data":"b0233b77826167f044d3183010b2b2c843853795f2e3a781543254b20a68fa3e"} Jul 31 14:48:38.308098 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:38.308025 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:38.367146 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:38.367114 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:38.681852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:38.681820 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:38.770311 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:38.770288 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:38.770452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:38.770384 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:38.770452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:38.770424 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:38.770452 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:38.770425 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:38.770588 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:38.770482 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:38.770645 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:38.770589 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:38.788355 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:38.788336 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:39.896779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:39.896750 2582 generic.go:356] "Generic (PLEG): container finished" podID="3d9ecebe-39b7-4f61-b0a3-aecbb90a9553" containerID="b5212350bbd798ae7e7e64d6b9a51804ed6c2665ee70cad1f178eacb54ccf4c8" exitCode=0 Jul 31 14:48:39.897270 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:39.896794 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fbb" event={"ID":"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553","Type":"ContainerDied","Data":"b5212350bbd798ae7e7e64d6b9a51804ed6c2665ee70cad1f178eacb54ccf4c8"} Jul 31 14:48:40.770190 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:40.770153 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:40.770190 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:40.770193 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:40.770418 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:40.770277 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:40.770418 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:40.770316 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:40.770418 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:40.770370 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:40.770621 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:40.770593 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:42.559994 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.559962 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:42.560424 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.560122 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:42.560424 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.560123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:42.560424 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.560184 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:42.560424 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.560188 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs podName:b3394f34-2e60-4e63-aee0-84ed3f6dada9 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.560169851 +0000 UTC m=+66.161606176 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs") pod "network-metrics-daemon-thsf6" (UID: "b3394f34-2e60-4e63-aee0-84ed3f6dada9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:48:42.560424 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.560232 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret podName:fa25e8f1-1145-4595-ae2a-5733ae0fc8fb nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.56022067 +0000 UTC m=+66.161656984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret") pod "global-pull-secret-syncer-tx9n9" (UID: "fa25e8f1-1145-4595-ae2a-5733ae0fc8fb") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:48:42.660825 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.660797 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6krv\" (UniqueName: \"kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv\") pod \"network-check-target-l4wr5\" (UID: \"4722e594-f22e-47e7-bd14-62ecccd174b1\") " pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:42.660945 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.660924 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:48:42.660945 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.660947 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:48:42.661127 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.660960 2582 projected.go:196] Error preparing data for projected volume kube-api-access-z6krv for pod openshift-network-diagnostics/network-check-target-l4wr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:42.661127 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.661018 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv podName:4722e594-f22e-47e7-bd14-62ecccd174b1 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.661001105 +0000 UTC m=+66.262437437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6krv" (UniqueName: "kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv") pod "network-check-target-l4wr5" (UID: "4722e594-f22e-47e7-bd14-62ecccd174b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:48:42.772847 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.772819 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:42.772847 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.772843 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:42.773061 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.772844 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:42.773061 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.772949 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thsf6" podUID="b3394f34-2e60-4e63-aee0-84ed3f6dada9" Jul 31 14:48:42.773161 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.773057 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4wr5" podUID="4722e594-f22e-47e7-bd14-62ecccd174b1" Jul 31 14:48:42.773161 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:42.773148 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx9n9" podUID="fa25e8f1-1145-4595-ae2a-5733ae0fc8fb" Jul 31 14:48:42.795883 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.795858 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeReady" Jul 31 14:48:42.796004 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.795981 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 31 14:48:42.896729 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.896703 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4t89r"] Jul 31 14:48:42.901690 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.901661 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:42.904920 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.904898 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-js6qm"] Jul 31 14:48:42.906860 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.906841 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 31 14:48:42.907345 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.907325 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p6mpf\"" Jul 31 14:48:42.907430 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.907325 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 31 14:48:42.908647 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.908628 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-js6qm" Jul 31 14:48:42.915644 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.915622 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 31 14:48:42.915745 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.915652 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 31 14:48:42.915893 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.915876 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-x9kr2\"" Jul 31 14:48:42.918322 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.918302 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 31 14:48:42.923392 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.923372 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4t89r"] Jul 31 14:48:42.923962 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:42.923946 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-js6qm"] Jul 31 14:48:43.031452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.031412 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:43.064271 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.064244 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnnv\" (UniqueName: \"kubernetes.io/projected/212418e4-f847-4cb7-b293-d1d119969ab3-kube-api-access-rfnnv\") pod \"ingress-canary-js6qm\" (UID: \"212418e4-f847-4cb7-b293-d1d119969ab3\") " pod="openshift-ingress-canary/ingress-canary-js6qm" Jul 31 14:48:43.064391 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.064279 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f45764f-729c-43b1-8a4b-39ab982be14b-config-volume\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.064391 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.064322 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.064391 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.064367 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4kwj\" (UniqueName: \"kubernetes.io/projected/4f45764f-729c-43b1-8a4b-39ab982be14b-kube-api-access-w4kwj\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.064541 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.064473 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/212418e4-f847-4cb7-b293-d1d119969ab3-cert\") pod \"ingress-canary-js6qm\" (UID: \"212418e4-f847-4cb7-b293-d1d119969ab3\") " pod="openshift-ingress-canary/ingress-canary-js6qm" Jul 31 14:48:43.064617 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.064544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f45764f-729c-43b1-8a4b-39ab982be14b-tmp-dir\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.147687 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.147617 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:43.165573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.165532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/212418e4-f847-4cb7-b293-d1d119969ab3-cert\") pod \"ingress-canary-js6qm\" (UID: \"212418e4-f847-4cb7-b293-d1d119969ab3\") " pod="openshift-ingress-canary/ingress-canary-js6qm" Jul 31 14:48:43.165714 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.165597 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f45764f-729c-43b1-8a4b-39ab982be14b-tmp-dir\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.165714 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.165626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnnv\" (UniqueName: \"kubernetes.io/projected/212418e4-f847-4cb7-b293-d1d119969ab3-kube-api-access-rfnnv\") pod \"ingress-canary-js6qm\" (UID: \"212418e4-f847-4cb7-b293-d1d119969ab3\") " pod="openshift-ingress-canary/ingress-canary-js6qm" Jul 31 14:48:43.165714 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.165643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f45764f-729c-43b1-8a4b-39ab982be14b-config-volume\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.165714 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.165679 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.165714 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.165702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4kwj\" (UniqueName: \"kubernetes.io/projected/4f45764f-729c-43b1-8a4b-39ab982be14b-kube-api-access-w4kwj\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.165990 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:43.165799 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:48:43.165990 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:43.165858 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls podName:4f45764f-729c-43b1-8a4b-39ab982be14b nodeName:}" failed. No retries permitted until 2026-07-31 14:48:43.665840167 +0000 UTC m=+35.267276487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls") pod "dns-default-4t89r" (UID: "4f45764f-729c-43b1-8a4b-39ab982be14b") : secret "dns-default-metrics-tls" not found Jul 31 14:48:43.166097 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.166013 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f45764f-729c-43b1-8a4b-39ab982be14b-tmp-dir\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.166277 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.166258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f45764f-729c-43b1-8a4b-39ab982be14b-config-volume\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.170307 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.170283 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/212418e4-f847-4cb7-b293-d1d119969ab3-cert\") pod \"ingress-canary-js6qm\" (UID: \"212418e4-f847-4cb7-b293-d1d119969ab3\") " pod="openshift-ingress-canary/ingress-canary-js6qm" Jul 31 14:48:43.174968 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.174943 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4kwj\" (UniqueName: \"kubernetes.io/projected/4f45764f-729c-43b1-8a4b-39ab982be14b-kube-api-access-w4kwj\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.175357 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.175336 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnnv\" (UniqueName: \"kubernetes.io/projected/212418e4-f847-4cb7-b293-d1d119969ab3-kube-api-access-rfnnv\") pod \"ingress-canary-js6qm\" (UID: \"212418e4-f847-4cb7-b293-d1d119969ab3\") " pod="openshift-ingress-canary/ingress-canary-js6qm" Jul 31 14:48:43.220876 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.220848 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-js6qm" Jul 31 14:48:43.241294 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.241275 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:43.669998 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.669970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:43.670601 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:43.670124 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:48:43.670601 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:43.670190 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls podName:4f45764f-729c-43b1-8a4b-39ab982be14b nodeName:}" failed. No retries permitted until 2026-07-31 14:48:44.670176197 +0000 UTC m=+36.271612512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls") pod "dns-default-4t89r" (UID: "4f45764f-729c-43b1-8a4b-39ab982be14b") : secret "dns-default-metrics-tls" not found Jul 31 14:48:43.921714 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:43.921505 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-js6qm"] Jul 31 14:48:43.928339 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:43.928314 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212418e4_f847_4cb7_b293_d1d119969ab3.slice/crio-5b9b7b2630f49bd3a932cb68f57bd43ca77f2ae8ff3497aa9ecfea0601aa92f1 WatchSource:0}: Error finding container 5b9b7b2630f49bd3a932cb68f57bd43ca77f2ae8ff3497aa9ecfea0601aa92f1: Status 404 returned error can't find the container with id 5b9b7b2630f49bd3a932cb68f57bd43ca77f2ae8ff3497aa9ecfea0601aa92f1 Jul 31 14:48:44.677146 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.677112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:44.677622 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:44.677274 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:48:44.677622 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:44.677340 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls podName:4f45764f-729c-43b1-8a4b-39ab982be14b nodeName:}" failed. No retries permitted until 2026-07-31 14:48:46.677321139 +0000 UTC m=+38.278757460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls") pod "dns-default-4t89r" (UID: "4f45764f-729c-43b1-8a4b-39ab982be14b") : secret "dns-default-metrics-tls" not found Jul 31 14:48:44.769825 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.769794 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:48:44.769955 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.769794 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:48:44.770030 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.769794 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:48:44.774360 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.774324 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 14:48:44.774487 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.774409 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 14:48:44.775140 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.775121 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 14:48:44.775233 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.775119 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 14:48:44.775233 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.775207 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qnc55\"" Jul 31 14:48:44.775346 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.775268 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2kwb2\"" Jul 31 14:48:44.911971 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.911941 2582 generic.go:356] "Generic (PLEG): container finished" podID="3d9ecebe-39b7-4f61-b0a3-aecbb90a9553" containerID="de3e1e70c59028e68b303b0e057d991261c6b920ebc8289b99782082812716ef" exitCode=0 Jul 31 14:48:44.912125 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.912021 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fbb" event={"ID":"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553","Type":"ContainerDied","Data":"de3e1e70c59028e68b303b0e057d991261c6b920ebc8289b99782082812716ef"} Jul 31 14:48:44.913379 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:44.913353 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-js6qm" event={"ID":"212418e4-f847-4cb7-b293-d1d119969ab3","Type":"ContainerStarted","Data":"5b9b7b2630f49bd3a932cb68f57bd43ca77f2ae8ff3497aa9ecfea0601aa92f1"} Jul 31 14:48:45.882415 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:45.882381 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:45.919691 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:45.919659 2582 generic.go:356] "Generic (PLEG): container finished" podID="3d9ecebe-39b7-4f61-b0a3-aecbb90a9553" containerID="db0a6a51b1085e7fbc2c53626185209ef3049bcc33fe6940c14586ae0c05e39c" exitCode=0 Jul 31 14:48:45.919836 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:45.919729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fbb" event={"ID":"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553","Type":"ContainerDied","Data":"db0a6a51b1085e7fbc2c53626185209ef3049bcc33fe6940c14586ae0c05e39c"} Jul 31 14:48:45.950922 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:45.950893 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:46.010957 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:46.010937 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:46.690245 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:46.690211 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:46.690387 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:46.690331 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:48:46.690387 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:46.690384 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls podName:4f45764f-729c-43b1-8a4b-39ab982be14b nodeName:}" failed. No retries permitted until 2026-07-31 14:48:50.69037052 +0000 UTC m=+42.291806835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls") pod "dns-default-4t89r" (UID: "4f45764f-729c-43b1-8a4b-39ab982be14b") : secret "dns-default-metrics-tls" not found Jul 31 14:48:46.893755 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:46.893714 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:46.925285 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:46.925257 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84fbb" event={"ID":"3d9ecebe-39b7-4f61-b0a3-aecbb90a9553","Type":"ContainerStarted","Data":"ebe41784128a3dd8fb18f0ea427b90c74fedd884c873628321b305bd0bddd1dc"} Jul 31 14:48:46.926596 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:46.926573 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-js6qm" event={"ID":"212418e4-f847-4cb7-b293-d1d119969ab3","Type":"ContainerStarted","Data":"bbda86e3dfb67d01e37887b14ad0945179b5991da92ac7a4b6ce64fea08f3a21"} Jul 31 14:48:46.952186 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:46.952106 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-84fbb" podStartSLOduration=8.808874237 podStartE2EDuration="37.952090586s" totalImagesPullingTime="29.143216349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:46.950865452 +0000 UTC m=+38.552301789" watchObservedRunningTime="2026-07-31 14:48:46.952090586 +0000 UTC m=+38.553526914" Jul 31 14:48:46.965908 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:46.965883 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:46.973043 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:46.973006 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-js6qm" podStartSLOduration=2.716051083 podStartE2EDuration="4.972992264s" totalImagesPullingTime="2.256941181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:46.972088389 +0000 UTC m=+38.573524727" watchObservedRunningTime="2026-07-31 14:48:46.972992264 +0000 UTC m=+38.574428604" Jul 31 14:48:47.041188 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:47.041154 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:47.110612 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:47.110578 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:47.214055 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:47.214000 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:47.314203 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:47.314185 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:48.148520 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:48.148492 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:48.216513 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:48.216490 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:50.716421 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:50.716384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:50.716863 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:50.716511 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:48:50.716863 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:50.716584 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls podName:4f45764f-729c-43b1-8a4b-39ab982be14b nodeName:}" failed. No retries permitted until 2026-07-31 14:48:58.716552888 +0000 UTC m=+50.317989203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls") pod "dns-default-4t89r" (UID: "4f45764f-729c-43b1-8a4b-39ab982be14b") : secret "dns-default-metrics-tls" not found Jul 31 14:48:51.660251 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:51.660217 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:53.324932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.324898 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r"] Jul 31 14:48:53.358717 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.358689 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r"] Jul 31 14:48:53.358855 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.358778 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r" Jul 31 14:48:53.361443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.361421 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:48:53.361443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.361437 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kwpms\"" Jul 31 14:48:53.362175 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.362157 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 31 14:48:53.411547 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.411522 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk"] Jul 31 14:48:53.436447 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.436418 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv"] Jul 31 14:48:53.436659 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.436642 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk" Jul 31 14:48:53.439508 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.439492 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vhcj8\"" Jul 31 14:48:53.458506 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.458488 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk"] Jul 31 14:48:53.458600 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.458512 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv"] Jul 31 14:48:53.458600 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.458592 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.460911 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.460896 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:48:53.461113 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.461098 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 31 14:48:53.461180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.461117 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 31 14:48:53.461255 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.461240 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 31 14:48:53.461690 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.461677 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-85lcq\"" Jul 31 14:48:53.511782 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.511761 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6"] Jul 31 14:48:53.532435 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.532418 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-cd69d8f8b-nwszz"] Jul 31 14:48:53.532603 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.532582 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.534140 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.534117 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swcj8\" (UniqueName: \"kubernetes.io/projected/9ff0d6ed-27d1-40aa-81da-2618aae3aa57-kube-api-access-swcj8\") pod \"volume-data-source-validator-5b8776669-k246r\" (UID: \"9ff0d6ed-27d1-40aa-81da-2618aae3aa57\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r" Jul 31 14:48:53.535529 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.535511 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 31 14:48:53.535888 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.535873 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kdk6v\"" Jul 31 14:48:53.536005 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.535876 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 31 14:48:53.536408 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.536387 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:48:53.536505 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.536390 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 31 14:48:53.552526 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.552510 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b95cd7d58-7zbqp"] Jul 31 14:48:53.552646 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.552632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.554804 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.554787 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 31 14:48:53.555016 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.555002 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 31 14:48:53.555064 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.555038 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 31 14:48:53.555117 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.555104 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kbt4f\"" Jul 31 14:48:53.555167 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.555154 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 31 14:48:53.555421 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.555403 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 31 14:48:53.555489 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.555470 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 31 14:48:53.576721 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.576680 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56"] Jul 31 14:48:53.576812 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.576799 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.579128 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.579105 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 31 14:48:53.579128 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.579118 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 31 14:48:53.579267 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.579142 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-plk7v\"" Jul 31 14:48:53.579267 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.579158 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 31 14:48:53.587146 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.587130 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 31 14:48:53.592709 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.592692 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6"] Jul 31 14:48:53.592771 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.592716 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-cd69d8f8b-nwszz"] Jul 31 14:48:53.592771 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.592731 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b95cd7d58-7zbqp"] Jul 31 14:48:53.592771 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.592742 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56"] Jul 31 14:48:53.592870 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.592799 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:48:53.595489 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.595469 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 31 14:48:53.595609 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.595514 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:48:53.596065 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.596052 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tpgfh\"" Jul 31 14:48:53.596157 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.596139 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 31 14:48:53.634499 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.634476 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2efbf5-f313-46f3-895d-4185cc06f1b6-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lsrgv\" (UID: \"fc2efbf5-f313-46f3-895d-4185cc06f1b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.634609 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.634516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9b2f85e-c510-431a-b0c9-4ff7102c46ad-serving-cert\") pod \"service-ca-operator-784f97bcf6-ktqv6\" (UID: \"c9b2f85e-c510-431a-b0c9-4ff7102c46ad\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.634609 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.634535 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rbwb\" (UniqueName: \"kubernetes.io/projected/fc2efbf5-f313-46f3-895d-4185cc06f1b6-kube-api-access-4rbwb\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lsrgv\" (UID: \"fc2efbf5-f313-46f3-895d-4185cc06f1b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.634717 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.634613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq8sk\" (UniqueName: \"kubernetes.io/projected/223ba0c0-fe75-455a-a713-c724d74ade9b-kube-api-access-gq8sk\") pod \"network-check-source-65bdbbff45-6hfhk\" (UID: \"223ba0c0-fe75-455a-a713-c724d74ade9b\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk" Jul 31 14:48:53.634717 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.634646 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9b2f85e-c510-431a-b0c9-4ff7102c46ad-config\") pod \"service-ca-operator-784f97bcf6-ktqv6\" (UID: \"c9b2f85e-c510-431a-b0c9-4ff7102c46ad\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.634717 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.634701 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7b64\" (UniqueName: \"kubernetes.io/projected/c9b2f85e-c510-431a-b0c9-4ff7102c46ad-kube-api-access-r7b64\") pod \"service-ca-operator-784f97bcf6-ktqv6\" (UID: \"c9b2f85e-c510-431a-b0c9-4ff7102c46ad\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.634830 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.634752 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc2efbf5-f313-46f3-895d-4185cc06f1b6-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lsrgv\" (UID: \"fc2efbf5-f313-46f3-895d-4185cc06f1b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.634830 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.634789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swcj8\" (UniqueName: \"kubernetes.io/projected/9ff0d6ed-27d1-40aa-81da-2618aae3aa57-kube-api-access-swcj8\") pod \"volume-data-source-validator-5b8776669-k246r\" (UID: \"9ff0d6ed-27d1-40aa-81da-2618aae3aa57\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r" Jul 31 14:48:53.645952 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.645934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swcj8\" (UniqueName: \"kubernetes.io/projected/9ff0d6ed-27d1-40aa-81da-2618aae3aa57-kube-api-access-swcj8\") pod \"volume-data-source-validator-5b8776669-k246r\" (UID: \"9ff0d6ed-27d1-40aa-81da-2618aae3aa57\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r" Jul 31 14:48:53.667763 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.667744 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r" Jul 31 14:48:53.700430 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.700402 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:53.735856 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.735828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-image-registry-private-configuration\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.735959 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.735872 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:48:53.735959 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.735905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq8sk\" (UniqueName: \"kubernetes.io/projected/223ba0c0-fe75-455a-a713-c724d74ade9b-kube-api-access-gq8sk\") pod \"network-check-source-65bdbbff45-6hfhk\" (UID: \"223ba0c0-fe75-455a-a713-c724d74ade9b\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk" Jul 31 14:48:53.736065 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.735968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9b2f85e-c510-431a-b0c9-4ff7102c46ad-config\") pod \"service-ca-operator-784f97bcf6-ktqv6\" (UID: \"c9b2f85e-c510-431a-b0c9-4ff7102c46ad\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.736065 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.736163 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736063 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trpn6\" (UniqueName: \"kubernetes.io/projected/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-kube-api-access-trpn6\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:48:53.736163 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736118 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-default-certificate\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.736257 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vtj\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-kube-api-access-m7vtj\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.736257 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736212 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2efbf5-f313-46f3-895d-4185cc06f1b6-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lsrgv\" (UID: \"fc2efbf5-f313-46f3-895d-4185cc06f1b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.736257 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736242 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-installation-pull-secrets\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.736401 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9b2f85e-c510-431a-b0c9-4ff7102c46ad-serving-cert\") pod \"service-ca-operator-784f97bcf6-ktqv6\" (UID: \"c9b2f85e-c510-431a-b0c9-4ff7102c46ad\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.736401 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rbwb\" (UniqueName: \"kubernetes.io/projected/fc2efbf5-f313-46f3-895d-4185cc06f1b6-kube-api-access-4rbwb\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lsrgv\" (UID: \"fc2efbf5-f313-46f3-895d-4185cc06f1b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.736401 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b64\" (UniqueName: \"kubernetes.io/projected/c9b2f85e-c510-431a-b0c9-4ff7102c46ad-kube-api-access-r7b64\") pod \"service-ca-operator-784f97bcf6-ktqv6\" (UID: \"c9b2f85e-c510-431a-b0c9-4ff7102c46ad\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.736587 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736404 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-stats-auth\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.736587 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.736587 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736479 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc2efbf5-f313-46f3-895d-4185cc06f1b6-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lsrgv\" (UID: \"fc2efbf5-f313-46f3-895d-4185cc06f1b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.736587 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736507 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.736587 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736551 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-certificates\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.736832 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-trusted-ca\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.736832 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736628 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-bound-sa-token\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.736832 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736638 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9b2f85e-c510-431a-b0c9-4ff7102c46ad-config\") pod \"service-ca-operator-784f97bcf6-ktqv6\" (UID: \"c9b2f85e-c510-431a-b0c9-4ff7102c46ad\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.736832 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5l4\" (UniqueName: \"kubernetes.io/projected/277d6797-d76b-4075-9963-c88cecf9eacb-kube-api-access-7l5l4\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.737015 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.736841 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74dec41d-08e7-42fb-80b4-89bd71bcf40e-ca-trust-extracted\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.737539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.737515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc2efbf5-f313-46f3-895d-4185cc06f1b6-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lsrgv\" (UID: \"fc2efbf5-f313-46f3-895d-4185cc06f1b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.738970 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.738944 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9b2f85e-c510-431a-b0c9-4ff7102c46ad-serving-cert\") pod \"service-ca-operator-784f97bcf6-ktqv6\" (UID: \"c9b2f85e-c510-431a-b0c9-4ff7102c46ad\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.739205 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.739187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2efbf5-f313-46f3-895d-4185cc06f1b6-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lsrgv\" (UID: \"fc2efbf5-f313-46f3-895d-4185cc06f1b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.745628 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.745548 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq8sk\" (UniqueName: \"kubernetes.io/projected/223ba0c0-fe75-455a-a713-c724d74ade9b-kube-api-access-gq8sk\") pod \"network-check-source-65bdbbff45-6hfhk\" (UID: \"223ba0c0-fe75-455a-a713-c724d74ade9b\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk" Jul 31 14:48:53.745997 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.745973 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rbwb\" (UniqueName: \"kubernetes.io/projected/fc2efbf5-f313-46f3-895d-4185cc06f1b6-kube-api-access-4rbwb\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lsrgv\" (UID: \"fc2efbf5-f313-46f3-895d-4185cc06f1b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.746517 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.746492 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7b64\" (UniqueName: \"kubernetes.io/projected/c9b2f85e-c510-431a-b0c9-4ff7102c46ad-kube-api-access-r7b64\") pod \"service-ca-operator-784f97bcf6-ktqv6\" (UID: \"c9b2f85e-c510-431a-b0c9-4ff7102c46ad\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.761288 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.761269 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:53.766896 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.766873 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" Jul 31 14:48:53.808574 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.808540 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r"] Jul 31 14:48:53.811334 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:53.811311 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff0d6ed_27d1_40aa_81da_2618aae3aa57.slice/crio-3d2e09da448ed79aa020129522a0e1b21e891351645d09c7a1f779b0427842b1 WatchSource:0}: Error finding container 3d2e09da448ed79aa020129522a0e1b21e891351645d09c7a1f779b0427842b1: Status 404 returned error can't find the container with id 3d2e09da448ed79aa020129522a0e1b21e891351645d09c7a1f779b0427842b1 Jul 31 14:48:53.837258 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837203 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.837258 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837234 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.837416 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-certificates\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.837416 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-trusted-ca\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.837416 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-bound-sa-token\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.837416 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:53.837309 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:48:53.837416 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5l4\" (UniqueName: \"kubernetes.io/projected/277d6797-d76b-4075-9963-c88cecf9eacb-kube-api-access-7l5l4\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.837416 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837344 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74dec41d-08e7-42fb-80b4-89bd71bcf40e-ca-trust-extracted\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.837416 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:53.837375 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:54.337354036 +0000 UTC m=+45.938790371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : secret "router-metrics-certs-default" not found Jul 31 14:48:53.837768 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:53.837526 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:54.337506433 +0000 UTC m=+45.938942751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : configmap references non-existent config key: service-ca.crt Jul 31 14:48:53.837768 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-image-registry-private-configuration\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.837768 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:48:53.837768 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837707 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.837768 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpn6\" (UniqueName: \"kubernetes.io/projected/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-kube-api-access-trpn6\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:48:53.838056 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837772 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74dec41d-08e7-42fb-80b4-89bd71bcf40e-ca-trust-extracted\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.838056 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837787 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-default-certificate\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.838056 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vtj\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-kube-api-access-m7vtj\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.838056 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:53.837869 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 31 14:48:53.838056 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837873 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-certificates\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.838056 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:53.837917 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls podName:bc8265f6-c73c-4776-8bf9-2c01b7bcab5e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:54.337902282 +0000 UTC m=+45.939338612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-56k56" (UID: "bc8265f6-c73c-4776-8bf9-2c01b7bcab5e") : secret "samples-operator-tls" not found Jul 31 14:48:53.838056 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-installation-pull-secrets\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.838056 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.837986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-stats-auth\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.838449 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:53.838100 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:48:53.838449 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:53.838121 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b95cd7d58-7zbqp: secret "image-registry-tls" not found Jul 31 14:48:53.838449 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:53.838177 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls podName:74dec41d-08e7-42fb-80b4-89bd71bcf40e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:54.338162748 +0000 UTC m=+45.939599067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls") pod "image-registry-7b95cd7d58-7zbqp" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e") : secret "image-registry-tls" not found Jul 31 14:48:53.838449 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.838318 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-trusted-ca\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.840025 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.840003 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-image-registry-private-configuration\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.840450 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.840415 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-default-certificate\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.840741 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.840723 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-installation-pull-secrets\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.840821 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.840807 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-stats-auth\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.841587 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.841569 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" Jul 31 14:48:53.848007 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.847987 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trpn6\" (UniqueName: \"kubernetes.io/projected/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-kube-api-access-trpn6\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:48:53.848208 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.848186 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-bound-sa-token\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.848348 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.848331 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5l4\" (UniqueName: \"kubernetes.io/projected/277d6797-d76b-4075-9963-c88cecf9eacb-kube-api-access-7l5l4\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:53.849471 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.849451 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vtj\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-kube-api-access-m7vtj\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:53.929098 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.929052 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv"] Jul 31 14:48:53.933570 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:53.933531 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2efbf5_f313_46f3_895d_4185cc06f1b6.slice/crio-482bf11342c9cd526b87221c945ff3b429e765fbf7b812b4bd5fa42c80892bbd WatchSource:0}: Error finding container 482bf11342c9cd526b87221c945ff3b429e765fbf7b812b4bd5fa42c80892bbd: Status 404 returned error can't find the container with id 482bf11342c9cd526b87221c945ff3b429e765fbf7b812b4bd5fa42c80892bbd Jul 31 14:48:53.943191 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.943161 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r" event={"ID":"9ff0d6ed-27d1-40aa-81da-2618aae3aa57","Type":"ContainerStarted","Data":"3d2e09da448ed79aa020129522a0e1b21e891351645d09c7a1f779b0427842b1"} Jul 31 14:48:53.944308 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:53.944279 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" event={"ID":"fc2efbf5-f313-46f3-895d-4185cc06f1b6","Type":"ContainerStarted","Data":"482bf11342c9cd526b87221c945ff3b429e765fbf7b812b4bd5fa42c80892bbd"} Jul 31 14:48:54.012377 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:54.012344 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6"] Jul 31 14:48:54.015604 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:54.015578 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b2f85e_c510_431a_b0c9_4ff7102c46ad.slice/crio-e91a9b8aa1ba3d23e9832253347adb1ee36b1db60f9387cb0519853947d50cb2 WatchSource:0}: Error finding container e91a9b8aa1ba3d23e9832253347adb1ee36b1db60f9387cb0519853947d50cb2: Status 404 returned error can't find the container with id e91a9b8aa1ba3d23e9832253347adb1ee36b1db60f9387cb0519853947d50cb2 Jul 31 14:48:54.044778 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:54.044755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk" Jul 31 14:48:54.203670 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:54.203646 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk"] Jul 31 14:48:54.206494 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:48:54.206467 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223ba0c0_fe75_455a_a713_c724d74ade9b.slice/crio-5bafbdec589db396ab39e43dfacfb81261d708e77d80122e8ae686d84f1a0215 WatchSource:0}: Error finding container 5bafbdec589db396ab39e43dfacfb81261d708e77d80122e8ae686d84f1a0215: Status 404 returned error can't find the container with id 5bafbdec589db396ab39e43dfacfb81261d708e77d80122e8ae686d84f1a0215 Jul 31 14:48:54.342028 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:54.341968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:48:54.342028 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:54.342005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:54.342360 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:54.342091 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:54.342360 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:54.342120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:54.342360 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:54.342125 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 31 14:48:54.342360 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:54.342179 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:48:54.342360 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:54.342188 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls podName:bc8265f6-c73c-4776-8bf9-2c01b7bcab5e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:55.342170129 +0000 UTC m=+46.943606444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-56k56" (UID: "bc8265f6-c73c-4776-8bf9-2c01b7bcab5e") : secret "samples-operator-tls" not found Jul 31 14:48:54.342360 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:54.342189 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:48:54.342360 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:54.342194 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b95cd7d58-7zbqp: secret "image-registry-tls" not found Jul 31 14:48:54.342360 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:54.342231 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:55.342221569 +0000 UTC m=+46.943657894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : configmap references non-existent config key: service-ca.crt Jul 31 14:48:54.342360 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:54.342269 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:55.342257068 +0000 UTC m=+46.943693386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : secret "router-metrics-certs-default" not found Jul 31 14:48:54.342360 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:54.342281 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls podName:74dec41d-08e7-42fb-80b4-89bd71bcf40e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:55.342274249 +0000 UTC m=+46.943710564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls") pod "image-registry-7b95cd7d58-7zbqp" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e") : secret "image-registry-tls" not found Jul 31 14:48:54.949304 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:54.949263 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" event={"ID":"c9b2f85e-c510-431a-b0c9-4ff7102c46ad","Type":"ContainerStarted","Data":"e91a9b8aa1ba3d23e9832253347adb1ee36b1db60f9387cb0519853947d50cb2"} Jul 31 14:48:54.950815 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:54.950752 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk" event={"ID":"223ba0c0-fe75-455a-a713-c724d74ade9b","Type":"ContainerStarted","Data":"5bafbdec589db396ab39e43dfacfb81261d708e77d80122e8ae686d84f1a0215"} Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:55.351707 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:55.351763 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:55.351820 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:55.351854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:55.352028 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:55.352043 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b95cd7d58-7zbqp: secret "image-registry-tls" not found Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:55.352104 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls podName:74dec41d-08e7-42fb-80b4-89bd71bcf40e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.352085062 +0000 UTC m=+48.953521392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls") pod "image-registry-7b95cd7d58-7zbqp" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e") : secret "image-registry-tls" not found Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:55.352483 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:55.352531 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.352516006 +0000 UTC m=+48.953952328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : secret "router-metrics-certs-default" not found Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:55.352615 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.352603235 +0000 UTC m=+48.954039562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : configmap references non-existent config key: service-ca.crt Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:55.352679 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 31 14:48:55.352786 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:55.352711 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls podName:bc8265f6-c73c-4776-8bf9-2c01b7bcab5e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.352700766 +0000 UTC m=+48.954137097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-56k56" (UID: "bc8265f6-c73c-4776-8bf9-2c01b7bcab5e") : secret "samples-operator-tls" not found Jul 31 14:48:57.367803 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.367761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:57.367803 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.367808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:48:57.368291 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.367858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:48:57.368291 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.367881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:48:57.368291 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.367924 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:48:57.368291 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.368013 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.36799385 +0000 UTC m=+52.969430171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : secret "router-metrics-certs-default" not found Jul 31 14:48:57.368291 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.368028 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.368022505 +0000 UTC m=+52.969458820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : configmap references non-existent config key: service-ca.crt Jul 31 14:48:57.368291 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.368031 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:48:57.368291 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.368043 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b95cd7d58-7zbqp: secret "image-registry-tls" not found Jul 31 14:48:57.368291 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.368042 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 31 14:48:57.368291 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.368123 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls podName:74dec41d-08e7-42fb-80b4-89bd71bcf40e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.368111133 +0000 UTC m=+52.969547462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls") pod "image-registry-7b95cd7d58-7zbqp" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e") : secret "image-registry-tls" not found Jul 31 14:48:57.368291 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.368139 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls podName:bc8265f6-c73c-4776-8bf9-2c01b7bcab5e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.368130509 +0000 UTC m=+52.969566824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-56k56" (UID: "bc8265f6-c73c-4776-8bf9-2c01b7bcab5e") : secret "samples-operator-tls" not found Jul 31 14:48:57.391746 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.391718 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j656c"] Jul 31 14:48:57.414183 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.414151 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:57.419357 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.419326 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j656c"] Jul 31 14:48:57.419484 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.419453 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.424876 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.424855 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 31 14:48:57.425011 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.424991 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 31 14:48:57.425078 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.425060 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 31 14:48:57.425442 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.425427 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 31 14:48:57.425903 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.425882 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v2m8k\"" Jul 31 14:48:57.469035 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.469009 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e727b6d2-1d23-42c5-bc08-42926513ede8-data-volume\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.469158 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.469050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e727b6d2-1d23-42c5-bc08-42926513ede8-crio-socket\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.469158 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.469148 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jf6z\" (UniqueName: \"kubernetes.io/projected/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-api-access-4jf6z\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.469281 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.469192 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.469335 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.469276 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.477406 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.477373 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:57.570567 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.570519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jf6z\" (UniqueName: \"kubernetes.io/projected/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-api-access-4jf6z\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.570694 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.570581 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.570694 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.570657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.570807 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.570706 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:48:57.570807 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.570749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e727b6d2-1d23-42c5-bc08-42926513ede8-data-volume\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.570807 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.570780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e727b6d2-1d23-42c5-bc08-42926513ede8-crio-socket\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.570929 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.570834 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:58.070815153 +0000 UTC m=+49.672251469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : configmap "kube-rbac-proxy" not found Jul 31 14:48:57.570929 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.570831 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:48:57.570929 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:57.570874 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:58.070864486 +0000 UTC m=+49.672300814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : secret "insights-runtime-extractor-tls" not found Jul 31 14:48:57.571078 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.570951 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e727b6d2-1d23-42c5-bc08-42926513ede8-crio-socket\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.571134 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.571109 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e727b6d2-1d23-42c5-bc08-42926513ede8-data-volume\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:57.580234 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:57.580208 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jf6z\" (UniqueName: \"kubernetes.io/projected/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-api-access-4jf6z\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:58.074952 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.074912 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:58.075158 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.075028 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:58.075158 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:58.075049 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:48:58.075158 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:58.075135 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:59.075114977 +0000 UTC m=+50.676551305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : configmap "kube-rbac-proxy" not found Jul 31 14:48:58.075158 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:58.075142 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:48:58.075378 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:58.075180 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:59.075168986 +0000 UTC m=+50.676605305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : secret "insights-runtime-extractor-tls" not found Jul 31 14:48:58.606607 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.606578 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:58.780405 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.780363 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:48:58.780516 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:58.780493 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:48:58.780592 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:58.780571 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls podName:4f45764f-729c-43b1-8a4b-39ab982be14b nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.780538058 +0000 UTC m=+66.381974379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls") pod "dns-default-4t89r" (UID: "4f45764f-729c-43b1-8a4b-39ab982be14b") : secret "dns-default-metrics-tls" not found Jul 31 14:48:58.938710 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.938682 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tmrz2_a3b6fa83-155e-4597-a00a-6940a61cac29/dns-node-resolver/0.log" Jul 31 14:48:58.962738 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.962669 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" event={"ID":"fc2efbf5-f313-46f3-895d-4185cc06f1b6","Type":"ContainerStarted","Data":"078a948f29f31fb4f7143cb231bbe71e7309654f175f2cae1a2725f997186fb0"} Jul 31 14:48:58.964277 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.964250 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r" event={"ID":"9ff0d6ed-27d1-40aa-81da-2618aae3aa57","Type":"ContainerStarted","Data":"b0f7879760a13ab12aaac0bed11ff12facf844f98cd179cb8583214915e27012"} Jul 31 14:48:58.965863 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.965811 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" event={"ID":"c9b2f85e-c510-431a-b0c9-4ff7102c46ad","Type":"ContainerStarted","Data":"bbb8ee71dfa80276f7d2accf22eace90facaaabdb0eba6f03cb4b2fa050da4b9"} Jul 31 14:48:58.967420 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.967396 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk" event={"ID":"223ba0c0-fe75-455a-a713-c724d74ade9b","Type":"ContainerStarted","Data":"34de341c0f62ddffa5f98d2ca2512b845ce1c6db16a08c6fa1b54f5b1cddec24"} Jul 31 14:48:58.979896 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.979853 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" podStartSLOduration=1.224724976 podStartE2EDuration="5.979836196s" totalImagesPullingTime="4.75511122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:58.978540107 +0000 UTC m=+50.579976445" watchObservedRunningTime="2026-07-31 14:48:58.979836196 +0000 UTC m=+50.581272534" Jul 31 14:48:58.997973 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:58.997925 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-6hfhk" podStartSLOduration=1.5059640239999998 podStartE2EDuration="5.997908327s" totalImagesPullingTime="4.491944303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:58.99685848 +0000 UTC m=+50.598294818" watchObservedRunningTime="2026-07-31 14:48:58.997908327 +0000 UTC m=+50.599344646" Jul 31 14:48:59.033263 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:59.033220 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k246r" podStartSLOduration=1.22042655 podStartE2EDuration="6.03320488s" totalImagesPullingTime="4.81277833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:59.014248859 +0000 UTC m=+50.615685197" watchObservedRunningTime="2026-07-31 14:48:59.03320488 +0000 UTC m=+50.634641218" Jul 31 14:48:59.033380 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:59.033305 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" podStartSLOduration=1.359049414 podStartE2EDuration="6.033298887s" totalImagesPullingTime="4.674249473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:59.032325895 +0000 UTC m=+50.633762226" watchObservedRunningTime="2026-07-31 14:48:59.033298887 +0000 UTC m=+50.634735225" Jul 31 14:48:59.083443 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:59.083415 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:59.083580 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:59.083511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:48:59.083580 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:59.083548 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:48:59.083691 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:59.083635 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.083617326 +0000 UTC m=+52.685053657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : configmap "kube-rbac-proxy" not found Jul 31 14:48:59.083691 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:59.083651 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:48:59.083782 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:48:59.083717 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.08370159 +0000 UTC m=+52.685137918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : secret "insights-runtime-extractor-tls" not found Jul 31 14:48:59.131615 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:59.131589 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:59.229415 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:59.229360 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:59.705240 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:59.705199 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:59.800439 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:59.800415 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:48:59.890750 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:48:59.890722 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:00.148842 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:00.148815 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s7km4_45c05bd2-6b63-4a74-86a8-f5a43da4c960/node-ca/0.log" Jul 31 14:49:00.148982 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:00.148941 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:00.253010 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:00.252978 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:00.353612 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:00.353592 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:00.541691 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:00.541642 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-js6qm_212418e4-f847-4cb7-b293-d1d119969ab3/serve-healthcheck-canary/0.log" Jul 31 14:49:00.940885 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:00.940860 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:01.041361 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.041326 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:01.101843 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.101818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:49:01.101977 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.101879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:49:01.101977 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.101936 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:49:01.102085 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.101995 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:05.101978145 +0000 UTC m=+56.703414460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : configmap "kube-rbac-proxy" not found Jul 31 14:49:01.102085 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.102056 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:49:01.102179 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.102120 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:05.102102438 +0000 UTC m=+56.703538770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : secret "insights-runtime-extractor-tls" not found Jul 31 14:49:01.168658 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.168635 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:01.266713 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.266650 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:01.403783 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.403753 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:01.403911 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.403790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:01.403911 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.403901 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:49:01.404013 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.403920 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:49:09.403903088 +0000 UTC m=+61.005339403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : configmap references non-existent config key: service-ca.crt Jul 31 14:49:01.404013 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.403974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:49:01.404013 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.404002 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:49:09.403980475 +0000 UTC m=+61.005416793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : secret "router-metrics-certs-default" not found Jul 31 14:49:01.404182 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.404047 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 31 14:49:01.404182 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.404063 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:49:01.404182 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.404084 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls podName:bc8265f6-c73c-4776-8bf9-2c01b7bcab5e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:09.404073208 +0000 UTC m=+61.005509548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-56k56" (UID: "bc8265f6-c73c-4776-8bf9-2c01b7bcab5e") : secret "samples-operator-tls" not found Jul 31 14:49:01.404182 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.404130 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:49:01.404182 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.404139 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b95cd7d58-7zbqp: secret "image-registry-tls" not found Jul 31 14:49:01.404182 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:01.404170 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls podName:74dec41d-08e7-42fb-80b4-89bd71bcf40e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:09.404161492 +0000 UTC m=+61.005597808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls") pod "image-registry-7b95cd7d58-7zbqp" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e") : secret "image-registry-tls" not found Jul 31 14:49:01.591396 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.591333 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:01.646022 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.645999 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:01.873367 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.873313 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:01.966075 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:01.966047 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:02.173478 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:02.173422 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:02.274025 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:02.274005 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:03.565703 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:03.565671 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:04.054228 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:04.054198 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:04.149896 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:04.149867 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:04.802013 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:04.801986 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:05.133496 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:05.133467 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:49:05.133660 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:05.133600 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:49:05.133660 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:05.133641 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:49:05.133736 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:05.133672 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:49:05.133736 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:05.133710 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:13.133691901 +0000 UTC m=+64.735128228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : secret "insights-runtime-extractor-tls" not found Jul 31 14:49:05.133736 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:05.133725 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:13.133718628 +0000 UTC m=+64.735154942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : configmap "kube-rbac-proxy" not found Jul 31 14:49:07.400385 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:07.400351 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:07.649322 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:07.649296 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:07.862712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:07.862644 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:08.010411 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:08.010380 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:08.130794 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:08.130766 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:08.234526 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:08.234502 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:08.332751 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:08.332726 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:08.388841 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:08.388790 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:08.922031 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:08.922008 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pdzx" Jul 31 14:49:08.958301 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:08.958280 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:09.466315 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.466290 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:49:09.466456 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.466321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:49:09.466456 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.466384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:09.466456 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.466403 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:09.466549 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:09.466493 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:49:09.466549 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:09.466505 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:49:09.466549 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:09.466512 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b95cd7d58-7zbqp: secret "image-registry-tls" not found Jul 31 14:49:09.466549 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:09.466528 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:49:25.46650926 +0000 UTC m=+77.067945587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : configmap references non-existent config key: service-ca.crt Jul 31 14:49:09.466694 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:09.466574 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls podName:74dec41d-08e7-42fb-80b4-89bd71bcf40e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:25.466543213 +0000 UTC m=+77.067979528 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls") pod "image-registry-7b95cd7d58-7zbqp" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e") : secret "image-registry-tls" not found Jul 31 14:49:09.466694 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:09.466598 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs podName:277d6797-d76b-4075-9963-c88cecf9eacb nodeName:}" failed. No retries permitted until 2026-07-31 14:49:25.466588188 +0000 UTC m=+77.068024506 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs") pod "router-default-cd69d8f8b-nwszz" (UID: "277d6797-d76b-4075-9963-c88cecf9eacb") : secret "router-metrics-certs-default" not found Jul 31 14:49:09.468633 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.468615 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc8265f6-c73c-4776-8bf9-2c01b7bcab5e-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-56k56\" (UID: \"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:49:09.502450 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.502430 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tpgfh\"" Jul 31 14:49:09.510408 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.510390 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" Jul 31 14:49:09.626310 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.626284 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56"] Jul 31 14:49:09.754436 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.754391 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:09.954321 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.954298 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:09.993262 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:09.993238 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" event={"ID":"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e","Type":"ContainerStarted","Data":"ba2f843afcc267848868f68a80cfdb63b7fc27d5c188bb2ed1233b3d317c5d8b"} Jul 31 14:49:10.359734 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:10.359701 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:10.528859 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:10.528829 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:10.586754 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:10.586732 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:10.947989 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:10.947950 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:11.013489 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:11.013461 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:11.420950 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:11.420919 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:11.521827 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:11.521766 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:11.740779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:11.740752 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:11.945724 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:11.945696 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:12.001685 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:12.001661 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" event={"ID":"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e","Type":"ContainerStarted","Data":"7e5f8b0ed8909372d121694b3145d27579b783e7cf315bf55af3d0d81c6d5f6f"} Jul 31 14:49:12.001822 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:12.001735 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" event={"ID":"bc8265f6-c73c-4776-8bf9-2c01b7bcab5e","Type":"ContainerStarted","Data":"0310cba6b72705c4f6fbcb260866e23c0f710f7028bf8c7cbe4220682d890bf9"} Jul 31 14:49:12.019843 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:12.019798 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-56k56" podStartSLOduration=17.527242969 podStartE2EDuration="19.019782582s" totalImagesPullingTime="1.492539613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:12.018968452 +0000 UTC m=+63.620404787" watchObservedRunningTime="2026-07-31 14:49:12.019782582 +0000 UTC m=+63.621218919" Jul 31 14:49:12.141744 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:12.141702 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:12.200915 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:12.200844 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:12.541970 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:12.541877 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:13.199145 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:13.199108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:49:13.199500 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:13.199170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:49:13.199500 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:13.199260 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:49:13.199500 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:13.199336 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls podName:e727b6d2-1d23-42c5-bc08-42926513ede8 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.199315835 +0000 UTC m=+80.800752150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j656c" (UID: "e727b6d2-1d23-42c5-bc08-42926513ede8") : secret "insights-runtime-extractor-tls" not found Jul 31 14:49:13.199779 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:13.199756 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e727b6d2-1d23-42c5-bc08-42926513ede8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:49:13.337144 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:13.337114 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:13.539729 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:13.539649 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:13.980644 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:13.980611 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:14.078826 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.078792 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:14.176471 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.176443 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:14.609920 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.609890 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:49:14.610268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.610051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:49:14.613719 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.613698 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 14:49:14.613815 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.613747 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 14:49:14.620937 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:14.620919 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 14:49:14.621009 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:14.620980 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs podName:b3394f34-2e60-4e63-aee0-84ed3f6dada9 nodeName:}" failed. No retries permitted until 2026-07-31 14:50:18.620965927 +0000 UTC m=+130.222402246 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs") pod "network-metrics-daemon-thsf6" (UID: "b3394f34-2e60-4e63-aee0-84ed3f6dada9") : secret "metrics-daemon-secret" not found Jul 31 14:49:14.622875 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.622853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa25e8f1-1145-4595-ae2a-5733ae0fc8fb-original-pull-secret\") pod \"global-pull-secret-syncer-tx9n9\" (UID: \"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb\") " pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:49:14.710996 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.710973 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6krv\" (UniqueName: \"kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv\") pod \"network-check-target-l4wr5\" (UID: \"4722e594-f22e-47e7-bd14-62ecccd174b1\") " pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:49:14.713153 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.713127 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6krv\" (UniqueName: \"kubernetes.io/projected/4722e594-f22e-47e7-bd14-62ecccd174b1-kube-api-access-z6krv\") pod \"network-check-target-l4wr5\" (UID: \"4722e594-f22e-47e7-bd14-62ecccd174b1\") " pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:49:14.763993 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.763964 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:14.785335 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.785318 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2kwb2\"" Jul 31 14:49:14.790277 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.790266 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx9n9" Jul 31 14:49:14.793409 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.793388 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:49:14.811242 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.811219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:49:14.813295 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.813277 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f45764f-729c-43b1-8a4b-39ab982be14b-metrics-tls\") pod \"dns-default-4t89r\" (UID: \"4f45764f-729c-43b1-8a4b-39ab982be14b\") " pod="openshift-dns/dns-default-4t89r" Jul 31 14:49:14.940199 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.940172 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tx9n9"] Jul 31 14:49:14.942309 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:14.942282 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa25e8f1_1145_4595_ae2a_5733ae0fc8fb.slice/crio-d811b1e80edf252553741d81c88137ad34fdd91f67c56ad241e230b1d3c75a9f WatchSource:0}: Error finding container d811b1e80edf252553741d81c88137ad34fdd91f67c56ad241e230b1d3c75a9f: Status 404 returned error can't find the container with id d811b1e80edf252553741d81c88137ad34fdd91f67c56ad241e230b1d3c75a9f Jul 31 14:49:14.960962 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:14.960941 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l4wr5"] Jul 31 14:49:14.963363 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:14.963339 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4722e594_f22e_47e7_bd14_62ecccd174b1.slice/crio-dc607d4950b1858b0a04127d9678f14f2214a13f3ad22e85f09300bd7802d6d2 WatchSource:0}: Error finding container dc607d4950b1858b0a04127d9678f14f2214a13f3ad22e85f09300bd7802d6d2: Status 404 returned error can't find the container with id dc607d4950b1858b0a04127d9678f14f2214a13f3ad22e85f09300bd7802d6d2 Jul 31 14:49:15.015823 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:15.015798 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tx9n9" event={"ID":"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb","Type":"ContainerStarted","Data":"d811b1e80edf252553741d81c88137ad34fdd91f67c56ad241e230b1d3c75a9f"} Jul 31 14:49:15.016788 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:15.016772 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p6mpf\"" Jul 31 14:49:15.016885 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:15.016863 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l4wr5" event={"ID":"4722e594-f22e-47e7-bd14-62ecccd174b1","Type":"ContainerStarted","Data":"dc607d4950b1858b0a04127d9678f14f2214a13f3ad22e85f09300bd7802d6d2"} Jul 31 14:49:15.024131 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:15.024111 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4t89r" Jul 31 14:49:15.147101 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:15.147019 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:15.155588 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:15.155569 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4t89r"] Jul 31 14:49:15.157388 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:15.157360 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f45764f_729c_43b1_8a4b_39ab982be14b.slice/crio-ec4bf2cdd11a5d227dd9446061702434678fea6a32e62cfa95dd80ebe5b24f94 WatchSource:0}: Error finding container ec4bf2cdd11a5d227dd9446061702434678fea6a32e62cfa95dd80ebe5b24f94: Status 404 returned error can't find the container with id ec4bf2cdd11a5d227dd9446061702434678fea6a32e62cfa95dd80ebe5b24f94 Jul 31 14:49:15.560414 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:15.560343 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:15.946181 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:15.946147 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:16.021770 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:16.021736 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4t89r" event={"ID":"4f45764f-729c-43b1-8a4b-39ab982be14b","Type":"ContainerStarted","Data":"ec4bf2cdd11a5d227dd9446061702434678fea6a32e62cfa95dd80ebe5b24f94"} Jul 31 14:49:16.023746 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:16.023719 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l4wr5" event={"ID":"4722e594-f22e-47e7-bd14-62ecccd174b1","Type":"ContainerStarted","Data":"16d86564b0181a637896ad3f55d02d62a5d0ffe0bb5f5575051bd774e0339118"} Jul 31 14:49:16.023964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:16.023941 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:49:16.056604 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:16.056538 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l4wr5" podStartSLOduration=67.056523279 podStartE2EDuration="1m7.056523279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:16.055201575 +0000 UTC m=+67.656637913" watchObservedRunningTime="2026-07-31 14:49:16.056523279 +0000 UTC m=+67.657959617" Jul 31 14:49:16.349724 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:16.349644 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:16.745635 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:16.745601 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:16.972768 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:16.972720 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:17.352987 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:17.352943 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:17.942294 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:17.942263 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:18.032777 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:18.032746 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4t89r" event={"ID":"4f45764f-729c-43b1-8a4b-39ab982be14b","Type":"ContainerStarted","Data":"578ce96a7f9bcbfaa0adee4ddb4d1aefa0a2fcdfe40bbf442f96d0731b1d41f7"} Jul 31 14:49:18.173993 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:18.173960 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:18.340435 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:18.340306 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:19.037725 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:19.037633 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tx9n9" event={"ID":"fa25e8f1-1145-4595-ae2a-5733ae0fc8fb","Type":"ContainerStarted","Data":"ec68c0c53bcc5a4de64f58f00b90a8b6bd2426f690e3780a30db8a9f9e09fe8f"} Jul 31 14:49:19.039151 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:19.039126 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4t89r" event={"ID":"4f45764f-729c-43b1-8a4b-39ab982be14b","Type":"ContainerStarted","Data":"8bbf87f72702ae01ebeb3af2506520889e0b952be57c272e8151f3d3cb0a53d8"} Jul 31 14:49:19.039288 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:19.039269 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4t89r" Jul 31 14:49:19.057602 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:19.057538 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tx9n9" podStartSLOduration=66.269519643 podStartE2EDuration="1m10.05752667s" totalImagesPullingTime="3.788007027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:19.056441746 +0000 UTC m=+70.657878082" watchObservedRunningTime="2026-07-31 14:49:19.05752667 +0000 UTC m=+70.658963008" Jul 31 14:49:19.074673 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:19.074632 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4t89r" podStartSLOduration=34.982443713 podStartE2EDuration="37.074621029s" totalImagesPullingTime="2.092177316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:19.073595279 +0000 UTC m=+70.675031616" watchObservedRunningTime="2026-07-31 14:49:19.074621029 +0000 UTC m=+70.676057361" Jul 31 14:49:19.626836 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:19.626799 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:19.689932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:19.689901 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:23.140535 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:23.140500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:25.490702 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.490665 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:25.490702 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.490705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:25.491253 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.490759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:49:25.491505 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.491485 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277d6797-d76b-4075-9963-c88cecf9eacb-service-ca-bundle\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:25.493237 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.493213 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") pod \"image-registry-7b95cd7d58-7zbqp\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:49:25.493345 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.493278 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277d6797-d76b-4075-9963-c88cecf9eacb-metrics-certs\") pod \"router-default-cd69d8f8b-nwszz\" (UID: \"277d6797-d76b-4075-9963-c88cecf9eacb\") " pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:25.604856 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.604829 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b95cd7d58-7zbqp"] Jul 31 14:49:25.609063 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.609047 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-plk7v\"" Jul 31 14:49:25.615329 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.615312 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:49:25.664175 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.664150 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kbt4f\"" Jul 31 14:49:25.672237 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.672215 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:25.712297 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.712256 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75cbcffcc4-vwhhd"] Jul 31 14:49:25.716832 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.716809 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.729611 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.729570 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75cbcffcc4-vwhhd"] Jul 31 14:49:25.751332 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.751158 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:25.752143 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.752120 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b95cd7d58-7zbqp"] Jul 31 14:49:25.754206 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:25.754177 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74dec41d_08e7_42fb_80b4_89bd71bcf40e.slice/crio-97c7e02426a2aaa18c733ec20eec1e00a881e397dd6074cfc8a5a45c41de037b WatchSource:0}: Error finding container 97c7e02426a2aaa18c733ec20eec1e00a881e397dd6074cfc8a5a45c41de037b: Status 404 returned error can't find the container with id 97c7e02426a2aaa18c733ec20eec1e00a881e397dd6074cfc8a5a45c41de037b Jul 31 14:49:25.793501 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.793478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f6b0388-8dfb-4109-be06-4df54eba0704-ca-trust-extracted\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.793687 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.793508 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkmvr\" (UniqueName: \"kubernetes.io/projected/2f6b0388-8dfb-4109-be06-4df54eba0704-kube-api-access-lkmvr\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.793687 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.793530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f6b0388-8dfb-4109-be06-4df54eba0704-image-registry-private-configuration\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.793687 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.793579 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f6b0388-8dfb-4109-be06-4df54eba0704-registry-tls\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.793687 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.793622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f6b0388-8dfb-4109-be06-4df54eba0704-installation-pull-secrets\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.793687 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.793681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f6b0388-8dfb-4109-be06-4df54eba0704-bound-sa-token\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.793871 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.793727 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f6b0388-8dfb-4109-be06-4df54eba0704-registry-certificates\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.793871 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.793743 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b0388-8dfb-4109-be06-4df54eba0704-trusted-ca\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.816167 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.816138 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-cd69d8f8b-nwszz"] Jul 31 14:49:25.820772 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:25.820741 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277d6797_d76b_4075_9963_c88cecf9eacb.slice/crio-c3fa548d6654e25eef0dab06d9c2adcf006795777f8cce51cbb1e1eb0a0ab53a WatchSource:0}: Error finding container c3fa548d6654e25eef0dab06d9c2adcf006795777f8cce51cbb1e1eb0a0ab53a: Status 404 returned error can't find the container with id c3fa548d6654e25eef0dab06d9c2adcf006795777f8cce51cbb1e1eb0a0ab53a Jul 31 14:49:25.824032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.824010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:25.894474 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.894443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f6b0388-8dfb-4109-be06-4df54eba0704-image-registry-private-configuration\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.894623 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.894512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f6b0388-8dfb-4109-be06-4df54eba0704-registry-tls\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.894623 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.894537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f6b0388-8dfb-4109-be06-4df54eba0704-installation-pull-secrets\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.894623 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.894575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f6b0388-8dfb-4109-be06-4df54eba0704-bound-sa-token\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.895763 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.894851 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f6b0388-8dfb-4109-be06-4df54eba0704-registry-certificates\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.895763 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.894906 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b0388-8dfb-4109-be06-4df54eba0704-trusted-ca\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.895763 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.894971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f6b0388-8dfb-4109-be06-4df54eba0704-ca-trust-extracted\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.895763 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.895001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmvr\" (UniqueName: \"kubernetes.io/projected/2f6b0388-8dfb-4109-be06-4df54eba0704-kube-api-access-lkmvr\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.895763 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.895597 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f6b0388-8dfb-4109-be06-4df54eba0704-ca-trust-extracted\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.895763 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.895679 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f6b0388-8dfb-4109-be06-4df54eba0704-registry-certificates\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.896177 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.896152 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b0388-8dfb-4109-be06-4df54eba0704-trusted-ca\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.897582 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.897117 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f6b0388-8dfb-4109-be06-4df54eba0704-image-registry-private-configuration\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.897691 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.897628 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f6b0388-8dfb-4109-be06-4df54eba0704-installation-pull-secrets\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.898308 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.898286 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f6b0388-8dfb-4109-be06-4df54eba0704-registry-tls\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.906815 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.906792 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmvr\" (UniqueName: \"kubernetes.io/projected/2f6b0388-8dfb-4109-be06-4df54eba0704-kube-api-access-lkmvr\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:25.907710 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:25.907682 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f6b0388-8dfb-4109-be06-4df54eba0704-bound-sa-token\") pod \"image-registry-75cbcffcc4-vwhhd\" (UID: \"2f6b0388-8dfb-4109-be06-4df54eba0704\") " pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:26.026709 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.026640 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:26.064468 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.064411 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" event={"ID":"74dec41d-08e7-42fb-80b4-89bd71bcf40e","Type":"ContainerStarted","Data":"8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58"} Jul 31 14:49:26.064673 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.064482 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" event={"ID":"74dec41d-08e7-42fb-80b4-89bd71bcf40e","Type":"ContainerStarted","Data":"97c7e02426a2aaa18c733ec20eec1e00a881e397dd6074cfc8a5a45c41de037b"} Jul 31 14:49:26.065224 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.064798 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:49:26.067040 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.066950 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-cd69d8f8b-nwszz" event={"ID":"277d6797-d76b-4075-9963-c88cecf9eacb","Type":"ContainerStarted","Data":"ea382f89519d1a3997f3665cacf0711ee177d6c0074b0e30cd4cb0967d262f3b"} Jul 31 14:49:26.067040 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.066981 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-cd69d8f8b-nwszz" event={"ID":"277d6797-d76b-4075-9963-c88cecf9eacb","Type":"ContainerStarted","Data":"c3fa548d6654e25eef0dab06d9c2adcf006795777f8cce51cbb1e1eb0a0ab53a"} Jul 31 14:49:26.124386 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.124340 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" podStartSLOduration=33.124326383 podStartE2EDuration="33.124326383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:26.12394313 +0000 UTC m=+77.725379470" watchObservedRunningTime="2026-07-31 14:49:26.124326383 +0000 UTC m=+77.725762719" Jul 31 14:49:26.178115 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.178062 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-cd69d8f8b-nwszz" podStartSLOduration=33.178047401 podStartE2EDuration="33.178047401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:26.177501377 +0000 UTC m=+77.778937729" watchObservedRunningTime="2026-07-31 14:49:26.178047401 +0000 UTC m=+77.779483736" Jul 31 14:49:26.185726 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.185699 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75cbcffcc4-vwhhd"] Jul 31 14:49:26.188455 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:26.188428 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6b0388_8dfb_4109_be06_4df54eba0704.slice/crio-c60e621abf08a28ec507a49049f6464f23ed85663af8e6740092e91a458982a6 WatchSource:0}: Error finding container c60e621abf08a28ec507a49049f6464f23ed85663af8e6740092e91a458982a6: Status 404 returned error can't find the container with id c60e621abf08a28ec507a49049f6464f23ed85663af8e6740092e91a458982a6 Jul 31 14:49:26.673221 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.673189 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:26.676588 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:26.676546 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:27.071763 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:27.071681 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" event={"ID":"2f6b0388-8dfb-4109-be06-4df54eba0704","Type":"ContainerStarted","Data":"62c2baa999c62e2dc4f5d6db9911ed3d657ccc4acc842d4cc40608754cdf8cb5"} Jul 31 14:49:27.071763 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:27.071733 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" event={"ID":"2f6b0388-8dfb-4109-be06-4df54eba0704","Type":"ContainerStarted","Data":"c60e621abf08a28ec507a49049f6464f23ed85663af8e6740092e91a458982a6"} Jul 31 14:49:27.072052 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:27.072036 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:27.072116 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:27.072074 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:27.073084 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:27.073051 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-cd69d8f8b-nwszz" Jul 31 14:49:27.098997 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:27.098947 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" podStartSLOduration=2.098932942 podStartE2EDuration="2.098932942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:27.098523359 +0000 UTC m=+78.699959697" watchObservedRunningTime="2026-07-31 14:49:27.098932942 +0000 UTC m=+78.700369278" Jul 31 14:49:29.044339 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:29.044311 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4t89r" Jul 31 14:49:29.218762 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:29.218731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:49:29.220929 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:29.220902 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e727b6d2-1d23-42c5-bc08-42926513ede8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j656c\" (UID: \"e727b6d2-1d23-42c5-bc08-42926513ede8\") " pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:49:29.232528 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:29.232504 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v2m8k\"" Jul 31 14:49:29.240702 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:29.240688 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j656c" Jul 31 14:49:29.369013 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:29.368985 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j656c"] Jul 31 14:49:29.376066 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:29.376043 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode727b6d2_1d23_42c5_bc08_42926513ede8.slice/crio-7f066b9dd6950ac2df345ba36eba29fbd6e531a17bc0aca54a5444398ff19206 WatchSource:0}: Error finding container 7f066b9dd6950ac2df345ba36eba29fbd6e531a17bc0aca54a5444398ff19206: Status 404 returned error can't find the container with id 7f066b9dd6950ac2df345ba36eba29fbd6e531a17bc0aca54a5444398ff19206 Jul 31 14:49:30.084750 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:30.084718 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j656c" event={"ID":"e727b6d2-1d23-42c5-bc08-42926513ede8","Type":"ContainerStarted","Data":"2edf2e62388ff6d659abfe9838bb37a5a211468218decde9a15acf3336d114c3"} Jul 31 14:49:30.085115 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:30.084780 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j656c" event={"ID":"e727b6d2-1d23-42c5-bc08-42926513ede8","Type":"ContainerStarted","Data":"7f066b9dd6950ac2df345ba36eba29fbd6e531a17bc0aca54a5444398ff19206"} Jul 31 14:49:30.433614 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:30.433588 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:30.519906 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:30.519871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:30.623974 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:30.623947 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:30.853473 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:30.853389 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:30.949396 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:30.949342 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:31.006299 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:31.006276 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:31.088999 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:31.088977 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j656c" event={"ID":"e727b6d2-1d23-42c5-bc08-42926513ede8","Type":"ContainerStarted","Data":"6037d1ebd51f76066b606c8e93232e0e41c06c74cf9dd3dd0c814f1dafebd8df"} Jul 31 14:49:32.093457 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:32.093427 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j656c" event={"ID":"e727b6d2-1d23-42c5-bc08-42926513ede8","Type":"ContainerStarted","Data":"f38387f05653dfdfa945507a72fde9bea9c3cae18352f939256f197680082d65"} Jul 31 14:49:32.115047 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:32.114999 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j656c" podStartSLOduration=32.703951972 podStartE2EDuration="35.114987143s" totalImagesPullingTime="2.411035171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:32.114088168 +0000 UTC m=+83.715524494" watchObservedRunningTime="2026-07-31 14:49:32.114987143 +0000 UTC m=+83.716423479" Jul 31 14:49:34.043926 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:34.043890 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:34.233128 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:34.233096 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:34.526547 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:34.526518 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:34.720105 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:34.720079 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:35.663630 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:35.663596 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:35.738727 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:35.738695 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:36.045977 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:36.045914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:36.109288 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:36.109258 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:38.514604 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:38.514574 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:38.617962 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:38.617934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:38.718060 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:38.718036 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:38.818943 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:38.818877 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:39.146160 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:39.146134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:39.244823 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:39.244796 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:39.553419 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:39.553335 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:39.653604 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:39.653576 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:39.762521 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:39.762501 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:39.923482 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:39.923441 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:40.140422 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:40.140395 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:40.226913 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:40.226833 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:40.965067 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:40.965030 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:41.175877 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.175829 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:41.237083 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.237029 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2nvvb"] Jul 31 14:49:41.241425 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.241407 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.245572 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.245531 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 31 14:49:41.245572 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.245542 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 31 14:49:41.245768 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.245622 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 31 14:49:41.246240 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.246224 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5lzwk\"" Jul 31 14:49:41.246240 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.246236 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 31 14:49:41.246350 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.246237 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 31 14:49:41.246350 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.246239 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 31 14:49:41.248672 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.248655 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8"] Jul 31 14:49:41.251664 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.251648 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.253992 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.253975 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-w6l6l\"" Jul 31 14:49:41.254058 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.254017 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 31 14:49:41.254334 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.254319 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 31 14:49:41.269576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.266816 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8"] Jul 31 14:49:41.273070 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.273050 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:41.281335 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.281318 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x"] Jul 31 14:49:41.284407 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.284393 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.287069 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.287049 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-s7xhq\"" Jul 31 14:49:41.287520 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.287275 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 31 14:49:41.287902 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.287885 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 31 14:49:41.288401 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.288384 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 31 14:49:41.314141 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.314096 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x"] Jul 31 14:49:41.372391 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.372365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:41.404459 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.404598 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-textfile\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.404598 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404485 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmnll\" (UniqueName: \"kubernetes.io/projected/dbeb1b85-9613-4cee-a362-1e8f60971b25-kube-api-access-mmnll\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.404598 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404500 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb1b85-9613-4cee-a362-1e8f60971b25-metrics-client-ca\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.404598 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b3a87a3-c8fe-45af-a748-fc5a3924c643-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.404769 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-tls\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.404769 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5xxd\" (UniqueName: \"kubernetes.io/projected/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-api-access-h5xxd\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.404769 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404676 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.404769 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404709 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.404985 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404791 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b3a87a3-c8fe-45af-a748-fc5a3924c643-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.404985 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404820 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dbeb1b85-9613-4cee-a362-1e8f60971b25-root\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.404985 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.404985 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404873 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b3a87a3-c8fe-45af-a748-fc5a3924c643-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.404985 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbjk\" (UniqueName: \"kubernetes.io/projected/7b3a87a3-c8fe-45af-a748-fc5a3924c643-kube-api-access-brbjk\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.404985 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404947 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.405194 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.404989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-wtmp\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.405194 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.405026 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.405194 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.405061 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.405194 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.405088 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbeb1b85-9613-4cee-a362-1e8f60971b25-sys\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.477906 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.477883 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:41.506190 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.506190 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.506333 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b3a87a3-c8fe-45af-a748-fc5a3924c643-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.506333 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506216 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dbeb1b85-9613-4cee-a362-1e8f60971b25-root\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.506333 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506248 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.506333 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506303 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dbeb1b85-9613-4cee-a362-1e8f60971b25-root\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.506494 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b3a87a3-c8fe-45af-a748-fc5a3924c643-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.506494 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506383 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbjk\" (UniqueName: \"kubernetes.io/projected/7b3a87a3-c8fe-45af-a748-fc5a3924c643-kube-api-access-brbjk\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.506494 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.506494 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-wtmp\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.506494 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.506738 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.506738 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:41.506513 2582 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 31 14:49:41.506738 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbeb1b85-9613-4cee-a362-1e8f60971b25-sys\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.506738 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506581 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.506738 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:41.506606 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b3a87a3-c8fe-45af-a748-fc5a3924c643-openshift-state-metrics-tls podName:7b3a87a3-c8fe-45af-a748-fc5a3924c643 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:42.006585886 +0000 UTC m=+93.608022202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7b3a87a3-c8fe-45af-a748-fc5a3924c643-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-9rbf8" (UID: "7b3a87a3-c8fe-45af-a748-fc5a3924c643") : secret "openshift-state-metrics-tls" not found Jul 31 14:49:41.506738 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-textfile\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.506738 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmnll\" (UniqueName: \"kubernetes.io/projected/dbeb1b85-9613-4cee-a362-1e8f60971b25-kube-api-access-mmnll\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.506738 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506680 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.506738 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb1b85-9613-4cee-a362-1e8f60971b25-metrics-client-ca\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.506738 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506729 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b3a87a3-c8fe-45af-a748-fc5a3924c643-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.507244 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-tls\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.507244 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.506793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5xxd\" (UniqueName: \"kubernetes.io/projected/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-api-access-h5xxd\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.507244 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.507009 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.507244 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.507021 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-textfile\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.507244 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.507047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b3a87a3-c8fe-45af-a748-fc5a3924c643-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.507489 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.507313 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb1b85-9613-4cee-a362-1e8f60971b25-metrics-client-ca\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.507588 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.507546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-wtmp\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.507679 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.507631 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbeb1b85-9613-4cee-a362-1e8f60971b25-sys\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.507905 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.507882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.507995 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.507885 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.509050 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.509026 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.509223 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.509206 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.509573 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.509537 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dbeb1b85-9613-4cee-a362-1e8f60971b25-node-exporter-tls\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.509733 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.509717 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b3a87a3-c8fe-45af-a748-fc5a3924c643-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.509778 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.509761 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.538981 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.538960 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5xxd\" (UniqueName: \"kubernetes.io/projected/6a232bc6-eeec-4edb-ae6c-2e89a68a4044-kube-api-access-h5xxd\") pod \"kube-state-metrics-6bd9664c4-ks86x\" (UID: \"6a232bc6-eeec-4edb-ae6c-2e89a68a4044\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.540601 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.540579 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbjk\" (UniqueName: \"kubernetes.io/projected/7b3a87a3-c8fe-45af-a748-fc5a3924c643-kube-api-access-brbjk\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:41.540894 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.540875 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmnll\" (UniqueName: \"kubernetes.io/projected/dbeb1b85-9613-4cee-a362-1e8f60971b25-kube-api-access-mmnll\") pod \"node-exporter-2nvvb\" (UID: \"dbeb1b85-9613-4cee-a362-1e8f60971b25\") " pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.550813 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.550798 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2nvvb" Jul 31 14:49:41.561446 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:41.561423 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbeb1b85_9613_4cee_a362_1e8f60971b25.slice/crio-c3c741b39e9c0f2732d1f379fa587743b4da5104a5e994cb34d83c5a249ab7d6 WatchSource:0}: Error finding container c3c741b39e9c0f2732d1f379fa587743b4da5104a5e994cb34d83c5a249ab7d6: Status 404 returned error can't find the container with id c3c741b39e9c0f2732d1f379fa587743b4da5104a5e994cb34d83c5a249ab7d6 Jul 31 14:49:41.580180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.580157 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:41.598281 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.598256 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" Jul 31 14:49:41.677436 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.677395 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:41.721762 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:41.721742 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x"] Jul 31 14:49:41.723959 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:41.723936 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a232bc6_eeec_4edb_ae6c_2e89a68a4044.slice/crio-d42fba558b4a65514a9699632f070cd58c1cb3281f771797075021c0b6659754 WatchSource:0}: Error finding container d42fba558b4a65514a9699632f070cd58c1cb3281f771797075021c0b6659754: Status 404 returned error can't find the container with id d42fba558b4a65514a9699632f070cd58c1cb3281f771797075021c0b6659754 Jul 31 14:49:42.010727 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.010694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b3a87a3-c8fe-45af-a748-fc5a3924c643-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:42.013037 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.013012 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b3a87a3-c8fe-45af-a748-fc5a3924c643-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-9rbf8\" (UID: \"7b3a87a3-c8fe-45af-a748-fc5a3924c643\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:42.030682 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.030664 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:42.131117 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.131075 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:42.131622 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.131586 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" event={"ID":"6a232bc6-eeec-4edb-ae6c-2e89a68a4044","Type":"ContainerStarted","Data":"d42fba558b4a65514a9699632f070cd58c1cb3281f771797075021c0b6659754"} Jul 31 14:49:42.132962 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.132930 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nvvb" event={"ID":"dbeb1b85-9613-4cee-a362-1e8f60971b25","Type":"ContainerStarted","Data":"c3c741b39e9c0f2732d1f379fa587743b4da5104a5e994cb34d83c5a249ab7d6"} Jul 31 14:49:42.160149 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.160126 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" Jul 31 14:49:42.202288 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.202256 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:42.300392 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.300161 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8"] Jul 31 14:49:42.484194 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:42.484163 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3a87a3_c8fe_45af_a748_fc5a3924c643.slice/crio-05fe39b3b067b963dc7b0ce82710cc729881deda896eadfee0fcdea9d66b6e3f WatchSource:0}: Error finding container 05fe39b3b067b963dc7b0ce82710cc729881deda896eadfee0fcdea9d66b6e3f: Status 404 returned error can't find the container with id 05fe39b3b067b963dc7b0ce82710cc729881deda896eadfee0fcdea9d66b6e3f Jul 31 14:49:42.499649 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.499621 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:42.911652 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:42.911617 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:43.140690 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:43.140657 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" event={"ID":"6a232bc6-eeec-4edb-ae6c-2e89a68a4044","Type":"ContainerStarted","Data":"8aace41c6ce4bafba34470f9217070a8114f5df65349aef8b718a71022b25206"} Jul 31 14:49:43.142467 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:43.142445 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" event={"ID":"7b3a87a3-c8fe-45af-a748-fc5a3924c643","Type":"ContainerStarted","Data":"e75ae701422693c09128a500be64f7548eb9bc04d053f8bc232b43bf9aa2c7a2"} Jul 31 14:49:43.142584 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:43.142499 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" event={"ID":"7b3a87a3-c8fe-45af-a748-fc5a3924c643","Type":"ContainerStarted","Data":"bba6cb8dd896118cc940487574cb8a355c1278e29ff3485174073a5b8f5c1c8e"} Jul 31 14:49:43.142584 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:43.142530 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" event={"ID":"7b3a87a3-c8fe-45af-a748-fc5a3924c643","Type":"ContainerStarted","Data":"05fe39b3b067b963dc7b0ce82710cc729881deda896eadfee0fcdea9d66b6e3f"} Jul 31 14:49:43.144495 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:43.144474 2582 generic.go:356] "Generic (PLEG): container finished" podID="dbeb1b85-9613-4cee-a362-1e8f60971b25" containerID="709e0943eeb37959ef87a6e0cda9fde0cc3462769cc00ef51dd81e212eee7af7" exitCode=0 Jul 31 14:49:43.144593 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:43.144525 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nvvb" event={"ID":"dbeb1b85-9613-4cee-a362-1e8f60971b25","Type":"ContainerDied","Data":"709e0943eeb37959ef87a6e0cda9fde0cc3462769cc00ef51dd81e212eee7af7"} Jul 31 14:49:43.672370 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:43.672338 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:43.780126 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:43.780092 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:44.150877 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.150839 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nvvb" event={"ID":"dbeb1b85-9613-4cee-a362-1e8f60971b25","Type":"ContainerStarted","Data":"5648af09c4984e45c9dfaf26186cced0709ea261e1e0d0a375a4d137d449624a"} Jul 31 14:49:44.151349 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.150944 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nvvb" event={"ID":"dbeb1b85-9613-4cee-a362-1e8f60971b25","Type":"ContainerStarted","Data":"93a42029e5cecbe5f288c6b2f759ba27e65e8fbb18d0733b29865eb84cae2d44"} Jul 31 14:49:44.153445 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.153410 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" event={"ID":"6a232bc6-eeec-4edb-ae6c-2e89a68a4044","Type":"ContainerStarted","Data":"9a1390575bf04d12a4f6d6d5438b5c7cd266c7dbff2093efc2062508ecd0995b"} Jul 31 14:49:44.153592 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.153477 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" event={"ID":"6a232bc6-eeec-4edb-ae6c-2e89a68a4044","Type":"ContainerStarted","Data":"8401ab8a3b11a3a460594ddeb9d3b64b883b6c41035bc061c40e9f0891ce5596"} Jul 31 14:49:44.155606 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.155579 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" event={"ID":"7b3a87a3-c8fe-45af-a748-fc5a3924c643","Type":"ContainerStarted","Data":"38f43381aff2b5428757e48ffb3607da8fe43fe93685409875b4eb43d4f6c250"} Jul 31 14:49:44.171212 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.171186 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx"] Jul 31 14:49:44.174657 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.174640 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.177364 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.177347 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 31 14:49:44.177463 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.177391 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 31 14:49:44.177635 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.177620 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 31 14:49:44.178884 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.178866 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 31 14:49:44.179002 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.178985 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 31 14:49:44.179254 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.179234 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7tn7nl50qjibi\"" Jul 31 14:49:44.179333 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.179274 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k2l8m\"" Jul 31 14:49:44.184499 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.184464 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2nvvb" podStartSLOduration=2.156933887 podStartE2EDuration="3.18445309s" totalImagesPullingTime="950.187439ms" totalInitContainerRuntime="77.331764ms" isStatefulPod=true podCreationTimestamp="2026-07-31 14:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:44.184015519 +0000 UTC m=+95.785451857" watchObservedRunningTime="2026-07-31 14:49:44.18445309 +0000 UTC m=+95.785889425" Jul 31 14:49:44.192260 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.192238 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx"] Jul 31 14:49:44.209035 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.209000 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-9rbf8" podStartSLOduration=2.079191325 podStartE2EDuration="3.208985577s" totalImagesPullingTime="1.129794252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:44.207822827 +0000 UTC m=+95.809259163" watchObservedRunningTime="2026-07-31 14:49:44.208985577 +0000 UTC m=+95.810421905" Jul 31 14:49:44.241897 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.241877 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:44.309576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.309521 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-ks86x" podStartSLOduration=1.97006469 podStartE2EDuration="3.309508509s" totalImagesPullingTime="1.339443819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:44.298663528 +0000 UTC m=+95.900099863" watchObservedRunningTime="2026-07-31 14:49:44.309508509 +0000 UTC m=+95.910944844" Jul 31 14:49:44.335012 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.334993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.335120 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.335042 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw9tv\" (UniqueName: \"kubernetes.io/projected/a68e2a91-d93a-4869-b109-f8b569d7bf80-kube-api-access-rw9tv\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.335234 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.335212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.335292 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.335259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.335348 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.335297 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68e2a91-d93a-4869-b109-f8b569d7bf80-metrics-client-ca\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.335391 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.335367 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-grpc-tls\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.335431 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.335422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.335888 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.335867 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-tls\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.367359 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.367338 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:44.436508 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.436447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-tls\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.436508 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.436480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.436508 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.436500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw9tv\" (UniqueName: \"kubernetes.io/projected/a68e2a91-d93a-4869-b109-f8b569d7bf80-kube-api-access-rw9tv\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.436726 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.436529 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.436726 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.436552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.436726 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.436603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68e2a91-d93a-4869-b109-f8b569d7bf80-metrics-client-ca\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.436726 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.436636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-grpc-tls\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.436726 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.436678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.437538 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.437493 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68e2a91-d93a-4869-b109-f8b569d7bf80-metrics-client-ca\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.439177 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.439156 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.439444 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.439425 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-grpc-tls\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.439491 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.439468 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-tls\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.439533 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.439506 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.439591 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.439511 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.439591 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.439545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a68e2a91-d93a-4869-b109-f8b569d7bf80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.457675 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.457655 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw9tv\" (UniqueName: \"kubernetes.io/projected/a68e2a91-d93a-4869-b109-f8b569d7bf80-kube-api-access-rw9tv\") pod \"thanos-querier-8447f8c8d6-tnvlx\" (UID: \"a68e2a91-d93a-4869-b109-f8b569d7bf80\") " pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.484475 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.484457 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:44.528848 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.528771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:44.597272 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.597235 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:44.617649 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:44.617617 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx"] Jul 31 14:49:44.622160 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:44.622132 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68e2a91_d93a_4869_b109_f8b569d7bf80.slice/crio-56e1e7f16ae9630e7caa9c46529372c2e58e7cba2fdf2614a4e23d355856cacb WatchSource:0}: Error finding container 56e1e7f16ae9630e7caa9c46529372c2e58e7cba2fdf2614a4e23d355856cacb: Status 404 returned error can't find the container with id 56e1e7f16ae9630e7caa9c46529372c2e58e7cba2fdf2614a4e23d355856cacb Jul 31 14:49:45.161033 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.160992 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" event={"ID":"a68e2a91-d93a-4869-b109-f8b569d7bf80","Type":"ContainerStarted","Data":"56e1e7f16ae9630e7caa9c46529372c2e58e7cba2fdf2614a4e23d355856cacb"} Jul 31 14:49:45.691987 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.691957 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7584bc8c56-cnkqv"] Jul 31 14:49:45.695276 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.695258 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.698529 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.698502 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 31 14:49:45.698635 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.698571 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 31 14:49:45.698635 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.698622 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hh6r8\"" Jul 31 14:49:45.698755 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.698636 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 31 14:49:45.698755 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.698659 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 31 14:49:45.698755 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.698574 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-39udasgfd6quo\"" Jul 31 14:49:45.705586 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.705541 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7584bc8c56-cnkqv"] Jul 31 14:49:45.850606 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.850575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6de51fd1-134a-4066-a999-c62eb56cf797-audit-log\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.850769 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.850662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de51fd1-134a-4066-a999-c62eb56cf797-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.850769 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.850723 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6de51fd1-134a-4066-a999-c62eb56cf797-client-ca-bundle\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.850769 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.850756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6de51fd1-134a-4066-a999-c62eb56cf797-secret-metrics-server-client-certs\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.850945 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.850819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6de51fd1-134a-4066-a999-c62eb56cf797-secret-metrics-server-tls\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.850945 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.850860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6de51fd1-134a-4066-a999-c62eb56cf797-metrics-server-audit-profiles\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.851042 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.850972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nbl\" (UniqueName: \"kubernetes.io/projected/6de51fd1-134a-4066-a999-c62eb56cf797-kube-api-access-l4nbl\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.871778 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.871732 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:45.951817 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.951731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nbl\" (UniqueName: \"kubernetes.io/projected/6de51fd1-134a-4066-a999-c62eb56cf797-kube-api-access-l4nbl\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.951817 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.951790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6de51fd1-134a-4066-a999-c62eb56cf797-audit-log\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.952041 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.951826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de51fd1-134a-4066-a999-c62eb56cf797-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.952041 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.951865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6de51fd1-134a-4066-a999-c62eb56cf797-client-ca-bundle\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.952041 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.951893 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6de51fd1-134a-4066-a999-c62eb56cf797-secret-metrics-server-client-certs\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.952198 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.952094 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6de51fd1-134a-4066-a999-c62eb56cf797-secret-metrics-server-tls\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.952198 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.952137 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6de51fd1-134a-4066-a999-c62eb56cf797-metrics-server-audit-profiles\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.952374 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.952278 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6de51fd1-134a-4066-a999-c62eb56cf797-audit-log\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.952738 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.952713 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de51fd1-134a-4066-a999-c62eb56cf797-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.953368 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.953347 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6de51fd1-134a-4066-a999-c62eb56cf797-metrics-server-audit-profiles\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.954607 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.954552 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6de51fd1-134a-4066-a999-c62eb56cf797-secret-metrics-server-client-certs\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.955164 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.955127 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6de51fd1-134a-4066-a999-c62eb56cf797-client-ca-bundle\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.955253 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.955183 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6de51fd1-134a-4066-a999-c62eb56cf797-secret-metrics-server-tls\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.959657 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.959637 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nbl\" (UniqueName: \"kubernetes.io/projected/6de51fd1-134a-4066-a999-c62eb56cf797-kube-api-access-l4nbl\") pod \"metrics-server-7584bc8c56-cnkqv\" (UID: \"6de51fd1-134a-4066-a999-c62eb56cf797\") " pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:45.969269 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:45.969202 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:46.007396 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:46.007365 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:49:46.071444 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:46.071417 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:49:46.182724 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:46.178788 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:46.634191 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:46.634167 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7584bc8c56-cnkqv"] Jul 31 14:49:46.638270 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:46.638249 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de51fd1_134a_4066_a999_c62eb56cf797.slice/crio-ec31ff900030e4c0a6a8affff04c80c35b2b6650a0eb5355fd8390cb06d0773a WatchSource:0}: Error finding container ec31ff900030e4c0a6a8affff04c80c35b2b6650a0eb5355fd8390cb06d0773a: Status 404 returned error can't find the container with id ec31ff900030e4c0a6a8affff04c80c35b2b6650a0eb5355fd8390cb06d0773a Jul 31 14:49:47.030175 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:47.030142 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l4wr5" Jul 31 14:49:47.170618 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:47.170579 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" event={"ID":"a68e2a91-d93a-4869-b109-f8b569d7bf80","Type":"ContainerStarted","Data":"61375f6a35591a5c348a986cb466aa7d253f73133a6abdfa02abca2c4c98f904"} Jul 31 14:49:47.170776 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:47.170661 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" event={"ID":"a68e2a91-d93a-4869-b109-f8b569d7bf80","Type":"ContainerStarted","Data":"e080e1e93808fbf978d9a35c7d8eef63521996daf8dd5ba7a4e78ec37f006e1b"} Jul 31 14:49:47.170776 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:47.170700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" event={"ID":"a68e2a91-d93a-4869-b109-f8b569d7bf80","Type":"ContainerStarted","Data":"947736cfa97eeb9de6a0317e195d3adeff28bc7ccc183125ac66cada35e0310e"} Jul 31 14:49:47.171527 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:47.171505 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" event={"ID":"6de51fd1-134a-4066-a999-c62eb56cf797","Type":"ContainerStarted","Data":"ec31ff900030e4c0a6a8affff04c80c35b2b6650a0eb5355fd8390cb06d0773a"} Jul 31 14:49:47.581440 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:47.581404 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:48.079966 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:48.079940 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75cbcffcc4-vwhhd" Jul 31 14:49:48.178822 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:48.178789 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" event={"ID":"a68e2a91-d93a-4869-b109-f8b569d7bf80","Type":"ContainerStarted","Data":"4be702db079a81f50df480a29fac69f92834d2d5812777173aaedf7aca8c7236"} Jul 31 14:49:48.178976 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:48.178867 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" event={"ID":"a68e2a91-d93a-4869-b109-f8b569d7bf80","Type":"ContainerStarted","Data":"92ed85bd5d0cc70bf6e96b907f69aefca894c3c83bf8977fcf5133941593322a"} Jul 31 14:49:48.240907 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:48.240875 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:48.340710 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:48.340682 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:48.443780 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:48.443760 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:49.184951 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:49.184918 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" event={"ID":"6de51fd1-134a-4066-a999-c62eb56cf797","Type":"ContainerStarted","Data":"9c55e060726552d2526a7b4c3363cf5c7b78cb0617aa9d85fea89a1fcc5cc41b"} Jul 31 14:49:49.188124 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:49.188101 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" event={"ID":"a68e2a91-d93a-4869-b109-f8b569d7bf80","Type":"ContainerStarted","Data":"4c97e09725f2a08c0d699faa934856e379ce6ebb33e61f5d98c130aaa527af7a"} Jul 31 14:49:49.188373 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:49.188353 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:49.247493 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:49.247450 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" podStartSLOduration=2.129874299 podStartE2EDuration="4.247436047s" totalImagesPullingTime="2.117561748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:49.246263242 +0000 UTC m=+100.847699579" watchObservedRunningTime="2026-07-31 14:49:49.247436047 +0000 UTC m=+100.848872384" Jul 31 14:49:49.320641 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:49.320586 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" podStartSLOduration=2.372860972 podStartE2EDuration="5.320568427s" totalImagesPullingTime="2.947707455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:49.318544223 +0000 UTC m=+100.919980561" watchObservedRunningTime="2026-07-31 14:49:49.320568427 +0000 UTC m=+100.922004758" Jul 31 14:49:51.086288 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.086228 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" podUID="74dec41d-08e7-42fb-80b4-89bd71bcf40e" containerName="registry" containerID="cri-o://8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58" gracePeriod=30 Jul 31 14:49:51.316296 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.316275 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:49:51.398264 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.398237 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7vtj\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-kube-api-access-m7vtj\") pod \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " Jul 31 14:49:51.398394 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.398271 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-bound-sa-token\") pod \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " Jul 31 14:49:51.398394 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.398295 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74dec41d-08e7-42fb-80b4-89bd71bcf40e-ca-trust-extracted\") pod \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " Jul 31 14:49:51.398394 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.398355 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-trusted-ca\") pod \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " Jul 31 14:49:51.398394 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.398386 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-image-registry-private-configuration\") pod \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " Jul 31 14:49:51.398628 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.398436 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") pod \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " Jul 31 14:49:51.398628 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.398478 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-certificates\") pod \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " Jul 31 14:49:51.398628 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.398510 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-installation-pull-secrets\") pod \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\" (UID: \"74dec41d-08e7-42fb-80b4-89bd71bcf40e\") " Jul 31 14:49:51.398946 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.398889 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "74dec41d-08e7-42fb-80b4-89bd71bcf40e" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:49:51.399090 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.399060 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "74dec41d-08e7-42fb-80b4-89bd71bcf40e" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:49:51.400799 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.400770 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "74dec41d-08e7-42fb-80b4-89bd71bcf40e" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:49:51.400961 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.400943 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-kube-api-access-m7vtj" (OuterVolumeSpecName: "kube-api-access-m7vtj") pod "74dec41d-08e7-42fb-80b4-89bd71bcf40e" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e"). InnerVolumeSpecName "kube-api-access-m7vtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:49:51.400961 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.400946 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "74dec41d-08e7-42fb-80b4-89bd71bcf40e" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:49:51.401200 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.401182 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "74dec41d-08e7-42fb-80b4-89bd71bcf40e" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:49:51.401244 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.401204 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "74dec41d-08e7-42fb-80b4-89bd71bcf40e" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:49:51.406493 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.406469 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dec41d-08e7-42fb-80b4-89bd71bcf40e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "74dec41d-08e7-42fb-80b4-89bd71bcf40e" (UID: "74dec41d-08e7-42fb-80b4-89bd71bcf40e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:49:51.499712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.499692 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-certificates\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:49:51.499712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.499711 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-installation-pull-secrets\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:49:51.499823 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.499720 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7vtj\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-kube-api-access-m7vtj\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:49:51.499823 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.499730 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-bound-sa-token\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:49:51.499823 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.499739 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74dec41d-08e7-42fb-80b4-89bd71bcf40e-ca-trust-extracted\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:49:51.499823 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.499747 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74dec41d-08e7-42fb-80b4-89bd71bcf40e-trusted-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:49:51.499823 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.499756 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74dec41d-08e7-42fb-80b4-89bd71bcf40e-image-registry-private-configuration\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:49:51.499823 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:51.499764 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74dec41d-08e7-42fb-80b4-89bd71bcf40e-registry-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:49:52.200175 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.200147 2582 generic.go:356] "Generic (PLEG): container finished" podID="74dec41d-08e7-42fb-80b4-89bd71bcf40e" containerID="8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58" exitCode=0 Jul 31 14:49:52.200589 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.200190 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" Jul 31 14:49:52.200589 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.200197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" event={"ID":"74dec41d-08e7-42fb-80b4-89bd71bcf40e","Type":"ContainerDied","Data":"8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58"} Jul 31 14:49:52.200589 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.200237 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b95cd7d58-7zbqp" event={"ID":"74dec41d-08e7-42fb-80b4-89bd71bcf40e","Type":"ContainerDied","Data":"97c7e02426a2aaa18c733ec20eec1e00a881e397dd6074cfc8a5a45c41de037b"} Jul 31 14:49:52.200589 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.200256 2582 scope.go:117] "RemoveContainer" containerID="8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58" Jul 31 14:49:52.208963 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.208943 2582 scope.go:117] "RemoveContainer" containerID="8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58" Jul 31 14:49:52.209231 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:49:52.209202 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58\": container with ID starting with 8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58 not found: ID does not exist" containerID="8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58" Jul 31 14:49:52.209292 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.209239 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58"} err="failed to get container status \"8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58\": rpc error: code = NotFound desc = could not find container \"8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58\": container with ID starting with 8df09df6e7d9446dd22ecf58944e51a5669ce4ed1ac9e0c7dc68ee52cfc6ff58 not found: ID does not exist" Jul 31 14:49:52.220551 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.220528 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b95cd7d58-7zbqp"] Jul 31 14:49:52.224660 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.224640 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b95cd7d58-7zbqp"] Jul 31 14:49:52.337228 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.337206 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:52.774338 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:52.774308 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74dec41d-08e7-42fb-80b4-89bd71bcf40e" path="/var/lib/kubelet/pods/74dec41d-08e7-42fb-80b4-89bd71bcf40e/volumes" Jul 31 14:49:54.749893 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.749866 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79c66b9b6f-lk9qt"] Jul 31 14:49:54.750252 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.750172 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74dec41d-08e7-42fb-80b4-89bd71bcf40e" containerName="registry" Jul 31 14:49:54.750252 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.750182 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="74dec41d-08e7-42fb-80b4-89bd71bcf40e" containerName="registry" Jul 31 14:49:54.750252 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.750253 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="74dec41d-08e7-42fb-80b4-89bd71bcf40e" containerName="registry" Jul 31 14:49:54.754959 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.754941 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.757426 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.757406 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 31 14:49:54.759825 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.759802 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 31 14:49:54.759939 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.759845 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 31 14:49:54.759939 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.759882 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g2qkk\"" Jul 31 14:49:54.759939 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.759883 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 31 14:49:54.759939 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.759845 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 31 14:49:54.760144 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.760102 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 31 14:49:54.760269 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.760249 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 31 14:49:54.764075 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.764055 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 31 14:49:54.765797 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.765781 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79c66b9b6f-lk9qt"] Jul 31 14:49:54.825397 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.825368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq8nv\" (UniqueName: \"kubernetes.io/projected/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-kube-api-access-vq8nv\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.825521 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.825445 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-oauth-config\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.825521 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.825478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-serving-cert\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.825521 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.825500 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-trusted-ca-bundle\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.825672 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.825551 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-oauth-serving-cert\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.825672 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.825596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-service-ca\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.825672 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.825637 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-config\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.926142 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.926115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-oauth-serving-cert\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.926235 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.926149 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-service-ca\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.926235 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.926176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-config\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.926235 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.926196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq8nv\" (UniqueName: \"kubernetes.io/projected/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-kube-api-access-vq8nv\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.926366 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.926235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-oauth-config\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.926366 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.926254 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-serving-cert\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.926366 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.926267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-trusted-ca-bundle\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.926891 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.926865 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-service-ca\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.926979 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.926909 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-oauth-serving-cert\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.926979 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.926917 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-config\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.927086 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.927067 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-trusted-ca-bundle\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.928644 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.928625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-oauth-config\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.928782 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.928764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-serving-cert\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:54.935094 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:54.935072 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq8nv\" (UniqueName: \"kubernetes.io/projected/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-kube-api-access-vq8nv\") pod \"console-79c66b9b6f-lk9qt\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:55.065918 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:55.065863 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:49:55.197784 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:55.197752 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8447f8c8d6-tnvlx" Jul 31 14:49:55.219194 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:55.219171 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79c66b9b6f-lk9qt"] Jul 31 14:49:55.219988 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:49:55.219965 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78de2b84_ae9f_4d0e_8c71_423b4d013c6e.slice/crio-b15a65b4a811eae7978d35474f47d9e7f86ffe96d672ec3baa28f0d8c9f1b86e WatchSource:0}: Error finding container b15a65b4a811eae7978d35474f47d9e7f86ffe96d672ec3baa28f0d8c9f1b86e: Status 404 returned error can't find the container with id b15a65b4a811eae7978d35474f47d9e7f86ffe96d672ec3baa28f0d8c9f1b86e Jul 31 14:49:55.352080 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:55.352010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:55.426132 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:55.426103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:56.222049 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:56.222020 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c66b9b6f-lk9qt" event={"ID":"78de2b84-ae9f-4d0e-8c71-423b4d013c6e","Type":"ContainerStarted","Data":"b15a65b4a811eae7978d35474f47d9e7f86ffe96d672ec3baa28f0d8c9f1b86e"} Jul 31 14:49:56.710081 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:56.710044 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:56.805510 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:56.805475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:56.864811 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:56.864777 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:57.154973 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:57.154941 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:57.258656 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:57.258622 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:58.233294 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:58.233215 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c66b9b6f-lk9qt" event={"ID":"78de2b84-ae9f-4d0e-8c71-423b4d013c6e","Type":"ContainerStarted","Data":"ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824"} Jul 31 14:49:58.257010 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:58.256959 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79c66b9b6f-lk9qt" podStartSLOduration=1.579139053 podStartE2EDuration="4.256943981s" totalImagesPullingTime="2.677804928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:58.256117554 +0000 UTC m=+109.857553893" watchObservedRunningTime="2026-07-31 14:49:58.256943981 +0000 UTC m=+109.858380316" Jul 31 14:49:59.924342 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:59.924301 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:59.995448 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:49:59.995419 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:00.381515 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:00.381489 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:00.495077 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:00.495051 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:04.634336 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:04.634270 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:04.735266 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:04.735231 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:05.067074 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:05.066984 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:50:05.067074 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:05.067041 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:50:05.071603 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:05.071581 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:50:05.261319 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:05.261289 2582 generic.go:356] "Generic (PLEG): container finished" podID="c9b2f85e-c510-431a-b0c9-4ff7102c46ad" containerID="bbb8ee71dfa80276f7d2accf22eace90facaaabdb0eba6f03cb4b2fa050da4b9" exitCode=0 Jul 31 14:50:05.261454 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:05.261359 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" event={"ID":"c9b2f85e-c510-431a-b0c9-4ff7102c46ad","Type":"ContainerDied","Data":"bbb8ee71dfa80276f7d2accf22eace90facaaabdb0eba6f03cb4b2fa050da4b9"} Jul 31 14:50:05.261704 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:05.261680 2582 scope.go:117] "RemoveContainer" containerID="bbb8ee71dfa80276f7d2accf22eace90facaaabdb0eba6f03cb4b2fa050da4b9" Jul 31 14:50:05.265137 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:05.265121 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:50:05.482973 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:05.482930 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:05.581440 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:05.581410 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:05.640574 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:05.640537 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:06.007616 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:06.007586 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:50:06.007787 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:06.007670 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:50:06.266896 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:06.266819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ktqv6" event={"ID":"c9b2f85e-c510-431a-b0c9-4ff7102c46ad","Type":"ContainerStarted","Data":"fafdbed72e183e6b8c8059b21e8a9c2d42e3442410ec83d9e57584ad7489bbe5"} Jul 31 14:50:11.028859 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:11.028821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:11.130662 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:11.130634 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:18.710916 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:18.710879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:50:18.713162 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:18.713137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3394f34-2e60-4e63-aee0-84ed3f6dada9-metrics-certs\") pod \"network-metrics-daemon-thsf6\" (UID: \"b3394f34-2e60-4e63-aee0-84ed3f6dada9\") " pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:50:18.997724 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:18.997650 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qnc55\"" Jul 31 14:50:19.005788 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:19.005768 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thsf6" Jul 31 14:50:19.131139 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:19.131119 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-thsf6"] Jul 31 14:50:19.133344 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:50:19.133316 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3394f34_2e60_4e63_aee0_84ed3f6dada9.slice/crio-87fe63118e012c40bb5cd5f288387e9f57653ce6871f9bd923534968b6f13bcb WatchSource:0}: Error finding container 87fe63118e012c40bb5cd5f288387e9f57653ce6871f9bd923534968b6f13bcb: Status 404 returned error can't find the container with id 87fe63118e012c40bb5cd5f288387e9f57653ce6871f9bd923534968b6f13bcb Jul 31 14:50:19.321180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:19.321102 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-thsf6" event={"ID":"b3394f34-2e60-4e63-aee0-84ed3f6dada9","Type":"ContainerStarted","Data":"87fe63118e012c40bb5cd5f288387e9f57653ce6871f9bd923534968b6f13bcb"} Jul 31 14:50:19.786413 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:19.786376 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:19.851612 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:19.851577 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:20.326885 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:20.326817 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-thsf6" event={"ID":"b3394f34-2e60-4e63-aee0-84ed3f6dada9","Type":"ContainerStarted","Data":"c225800dcd3aa0358c7b5b17dc15dbb02a8deaf323efac55a5b88a3427a2e49e"} Jul 31 14:50:20.326885 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:20.326876 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-thsf6" event={"ID":"b3394f34-2e60-4e63-aee0-84ed3f6dada9","Type":"ContainerStarted","Data":"b14844e70b62d00dd3bbfd1410f8c27d78e6bfc3fbeb9174cb56acdb05d10c9f"} Jul 31 14:50:20.348180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:20.348138 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-thsf6" podStartSLOduration=130.48852883 podStartE2EDuration="2m11.348122226s" totalImagesPullingTime="859.593396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:20.347715654 +0000 UTC m=+131.949151997" watchObservedRunningTime="2026-07-31 14:50:20.348122226 +0000 UTC m=+131.949558561" Jul 31 14:50:20.447152 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:20.447122 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:20.507526 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:20.507504 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:25.348608 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:25.348572 2582 generic.go:356] "Generic (PLEG): container finished" podID="fc2efbf5-f313-46f3-895d-4185cc06f1b6" containerID="078a948f29f31fb4f7143cb231bbe71e7309654f175f2cae1a2725f997186fb0" exitCode=0 Jul 31 14:50:25.348988 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:25.348614 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" event={"ID":"fc2efbf5-f313-46f3-895d-4185cc06f1b6","Type":"ContainerDied","Data":"078a948f29f31fb4f7143cb231bbe71e7309654f175f2cae1a2725f997186fb0"} Jul 31 14:50:25.348988 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:25.348925 2582 scope.go:117] "RemoveContainer" containerID="078a948f29f31fb4f7143cb231bbe71e7309654f175f2cae1a2725f997186fb0" Jul 31 14:50:26.013193 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:26.013166 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:50:26.017228 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:26.017204 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7584bc8c56-cnkqv" Jul 31 14:50:26.355143 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:26.355069 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lsrgv" event={"ID":"fc2efbf5-f313-46f3-895d-4185cc06f1b6","Type":"ContainerStarted","Data":"73962536912a96a83d27fb4aa7f2b76bb2186e8e1ecdbe12ff252e634e17f883"} Jul 31 14:50:39.095356 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:39.095318 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:39.152685 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:50:39.152659 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:00.517742 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:00.517703 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:00.774588 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:00.774505 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:08.311796 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.311758 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7584c7fb9f-db426"] Jul 31 14:51:08.315176 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.315153 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.327765 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.327740 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7584c7fb9f-db426"] Jul 31 14:51:08.364527 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.364467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:08.388671 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.388646 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-trusted-ca-bundle\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.388761 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.388676 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbll\" (UniqueName: \"kubernetes.io/projected/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-kube-api-access-dxbll\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.388761 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.388710 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-oauth-config\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.388838 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.388789 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-serving-cert\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.388877 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.388845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-oauth-serving-cert\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.388916 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.388876 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-service-ca\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.388916 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.388903 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-config\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.442732 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.442706 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:08.489869 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.489846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-trusted-ca-bundle\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.489962 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.489874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbll\" (UniqueName: \"kubernetes.io/projected/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-kube-api-access-dxbll\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.489962 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.489908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-oauth-config\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.489962 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.489925 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-serving-cert\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.489962 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.489950 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-oauth-serving-cert\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.490162 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.489968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-service-ca\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.490162 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.489983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-config\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.490794 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.490774 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-oauth-serving-cert\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.490874 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.490781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-config\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.490971 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.490948 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-service-ca\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.491091 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.491074 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-trusted-ca-bundle\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.492312 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.492287 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-serving-cert\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.492398 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.492349 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-oauth-config\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.497705 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.497683 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbll\" (UniqueName: \"kubernetes.io/projected/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-kube-api-access-dxbll\") pod \"console-7584c7fb9f-db426\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.509422 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.509398 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:08.574285 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.574222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:08.623931 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.623908 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:08.759158 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:08.759131 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7584c7fb9f-db426"] Jul 31 14:51:08.761713 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:51:08.761682 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2dce68_d02f_49fc_b7f9_0b9f60a3123d.slice/crio-7581f096b1685edf40c03f0f4a681895a0ce82f4fcc47030805fc11129e5962e WatchSource:0}: Error finding container 7581f096b1685edf40c03f0f4a681895a0ce82f4fcc47030805fc11129e5962e: Status 404 returned error can't find the container with id 7581f096b1685edf40c03f0f4a681895a0ce82f4fcc47030805fc11129e5962e Jul 31 14:51:09.122440 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:09.122410 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:09.201079 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:09.201048 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:09.519933 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:09.519898 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7584c7fb9f-db426" event={"ID":"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d","Type":"ContainerStarted","Data":"4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e"} Jul 31 14:51:09.520401 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:09.519950 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7584c7fb9f-db426" event={"ID":"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d","Type":"ContainerStarted","Data":"7581f096b1685edf40c03f0f4a681895a0ce82f4fcc47030805fc11129e5962e"} Jul 31 14:51:09.539321 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:09.539268 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7584c7fb9f-db426" podStartSLOduration=1.539255233 podStartE2EDuration="1.539255233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:09.537657643 +0000 UTC m=+181.139093993" watchObservedRunningTime="2026-07-31 14:51:09.539255233 +0000 UTC m=+181.140691569" Jul 31 14:51:18.624423 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:18.624390 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:18.625021 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:18.624455 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:18.629030 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:18.629009 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:19.559859 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:19.559832 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:51:19.607255 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:19.607222 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79c66b9b6f-lk9qt"] Jul 31 14:51:19.720239 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:19.720208 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:19.824046 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:19.823983 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:19.921728 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:19.921697 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:28.483926 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:28.483889 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:44.627394 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.627327 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79c66b9b6f-lk9qt" podUID="78de2b84-ae9f-4d0e-8c71-423b4d013c6e" containerName="console" containerID="cri-o://ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824" gracePeriod=15 Jul 31 14:51:44.867784 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.867756 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:51:44.992224 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992180 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-service-ca\") pod \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " Jul 31 14:51:44.992224 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-oauth-serving-cert\") pod \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " Jul 31 14:51:44.992483 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992251 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-trusted-ca-bundle\") pod \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " Jul 31 14:51:44.992483 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992284 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-config\") pod \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " Jul 31 14:51:44.992483 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992310 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-serving-cert\") pod \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " Jul 31 14:51:44.992483 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992329 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq8nv\" (UniqueName: \"kubernetes.io/projected/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-kube-api-access-vq8nv\") pod \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " Jul 31 14:51:44.992483 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992360 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-oauth-config\") pod \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\" (UID: \"78de2b84-ae9f-4d0e-8c71-423b4d013c6e\") " Jul 31 14:51:44.992824 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992744 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "78de2b84-ae9f-4d0e-8c71-423b4d013c6e" (UID: "78de2b84-ae9f-4d0e-8c71-423b4d013c6e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:44.992824 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992787 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "78de2b84-ae9f-4d0e-8c71-423b4d013c6e" (UID: "78de2b84-ae9f-4d0e-8c71-423b4d013c6e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:44.992824 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992813 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-config" (OuterVolumeSpecName: "console-config") pod "78de2b84-ae9f-4d0e-8c71-423b4d013c6e" (UID: "78de2b84-ae9f-4d0e-8c71-423b4d013c6e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:44.992977 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.992824 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-service-ca" (OuterVolumeSpecName: "service-ca") pod "78de2b84-ae9f-4d0e-8c71-423b4d013c6e" (UID: "78de2b84-ae9f-4d0e-8c71-423b4d013c6e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:44.994743 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.994717 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "78de2b84-ae9f-4d0e-8c71-423b4d013c6e" (UID: "78de2b84-ae9f-4d0e-8c71-423b4d013c6e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:51:44.994743 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.994719 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-kube-api-access-vq8nv" (OuterVolumeSpecName: "kube-api-access-vq8nv") pod "78de2b84-ae9f-4d0e-8c71-423b4d013c6e" (UID: "78de2b84-ae9f-4d0e-8c71-423b4d013c6e"). InnerVolumeSpecName "kube-api-access-vq8nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:44.994873 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:44.994754 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "78de2b84-ae9f-4d0e-8c71-423b4d013c6e" (UID: "78de2b84-ae9f-4d0e-8c71-423b4d013c6e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:51:45.093196 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.093155 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:51:45.093196 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.093190 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-oauth-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:51:45.093196 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.093199 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-trusted-ca-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:51:45.093196 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.093208 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:51:45.093196 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.093216 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:51:45.093466 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.093224 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq8nv\" (UniqueName: \"kubernetes.io/projected/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-kube-api-access-vq8nv\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:51:45.093466 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.093234 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78de2b84-ae9f-4d0e-8c71-423b4d013c6e-console-oauth-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:51:45.654611 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.654576 2582 generic.go:356] "Generic (PLEG): container finished" podID="78de2b84-ae9f-4d0e-8c71-423b4d013c6e" containerID="ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824" exitCode=0 Jul 31 14:51:45.654611 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.654615 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c66b9b6f-lk9qt" Jul 31 14:51:45.655044 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.654649 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c66b9b6f-lk9qt" event={"ID":"78de2b84-ae9f-4d0e-8c71-423b4d013c6e","Type":"ContainerDied","Data":"ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824"} Jul 31 14:51:45.655044 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.654704 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c66b9b6f-lk9qt" event={"ID":"78de2b84-ae9f-4d0e-8c71-423b4d013c6e","Type":"ContainerDied","Data":"b15a65b4a811eae7978d35474f47d9e7f86ffe96d672ec3baa28f0d8c9f1b86e"} Jul 31 14:51:45.655044 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.654726 2582 scope.go:117] "RemoveContainer" containerID="ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824" Jul 31 14:51:45.663524 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.663509 2582 scope.go:117] "RemoveContainer" containerID="ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824" Jul 31 14:51:45.663821 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:51:45.663803 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824\": container with ID starting with ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824 not found: ID does not exist" containerID="ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824" Jul 31 14:51:45.663874 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.663830 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824"} err="failed to get container status \"ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824\": rpc error: code = NotFound desc = could not find container \"ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824\": container with ID starting with ae2763ff7b34f4ad69d4efd7d49fec5269957bdf53c86fce866355dac843b824 not found: ID does not exist" Jul 31 14:51:45.680840 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.680818 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79c66b9b6f-lk9qt"] Jul 31 14:51:45.683684 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:45.683666 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79c66b9b6f-lk9qt"] Jul 31 14:51:46.774605 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:46.774576 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78de2b84-ae9f-4d0e-8c71-423b4d013c6e" path="/var/lib/kubelet/pods/78de2b84-ae9f-4d0e-8c71-423b4d013c6e/volumes" Jul 31 14:51:47.810351 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:47.810319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:47.918466 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:51:47.918442 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:15.782481 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:15.782450 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:15.838445 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:15.838421 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:27.780701 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:27.780667 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:27.901074 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:27.901045 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:34.626019 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:34.625990 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:34.725372 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:34.725341 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:34.782164 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:34.782137 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:38.433921 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.433890 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c686965b-78r6c"] Jul 31 14:52:38.434266 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.434171 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78de2b84-ae9f-4d0e-8c71-423b4d013c6e" containerName="console" Jul 31 14:52:38.434266 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.434181 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="78de2b84-ae9f-4d0e-8c71-423b4d013c6e" containerName="console" Jul 31 14:52:38.434266 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.434248 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="78de2b84-ae9f-4d0e-8c71-423b4d013c6e" containerName="console" Jul 31 14:52:38.437096 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.437078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.450184 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.450160 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c686965b-78r6c"] Jul 31 14:52:38.525039 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.525014 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:38.579274 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.579247 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:38.589353 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.589328 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-oauth-serving-cert\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.589448 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.589364 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-serving-cert\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.589448 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.589381 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-config\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.589448 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.589405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-oauth-config\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.589448 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.589438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-service-ca\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.589610 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.589541 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p95r\" (UniqueName: \"kubernetes.io/projected/ec38bdde-60ff-4195-8da2-1dd8b81e514c-kube-api-access-9p95r\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.589610 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.589603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-trusted-ca-bundle\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.690230 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.690183 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p95r\" (UniqueName: \"kubernetes.io/projected/ec38bdde-60ff-4195-8da2-1dd8b81e514c-kube-api-access-9p95r\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.690230 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.690218 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-trusted-ca-bundle\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.690352 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.690239 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-oauth-serving-cert\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.690352 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.690264 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-serving-cert\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.690352 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.690279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-config\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.690352 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.690297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-oauth-config\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.690352 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.690311 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-service-ca\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.691050 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.691019 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-oauth-serving-cert\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.691050 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.691027 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-service-ca\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.691211 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.691087 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-config\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.691211 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.691109 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-trusted-ca-bundle\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.692846 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.692815 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-serving-cert\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.692922 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.692906 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-oauth-config\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.701517 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.701496 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p95r\" (UniqueName: \"kubernetes.io/projected/ec38bdde-60ff-4195-8da2-1dd8b81e514c-kube-api-access-9p95r\") pod \"console-5c686965b-78r6c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.746847 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.746816 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:38.879509 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:38.879481 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c686965b-78r6c"] Jul 31 14:52:38.882171 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:52:38.882146 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec38bdde_60ff_4195_8da2_1dd8b81e514c.slice/crio-858c38d2ca9df01852e6eae568272a6e4f1a492ef31a6b28300dbda9ee477927 WatchSource:0}: Error finding container 858c38d2ca9df01852e6eae568272a6e4f1a492ef31a6b28300dbda9ee477927: Status 404 returned error can't find the container with id 858c38d2ca9df01852e6eae568272a6e4f1a492ef31a6b28300dbda9ee477927 Jul 31 14:52:39.860754 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:39.860717 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c686965b-78r6c" event={"ID":"ec38bdde-60ff-4195-8da2-1dd8b81e514c","Type":"ContainerStarted","Data":"ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a"} Jul 31 14:52:39.861115 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:39.860799 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c686965b-78r6c" event={"ID":"ec38bdde-60ff-4195-8da2-1dd8b81e514c","Type":"ContainerStarted","Data":"858c38d2ca9df01852e6eae568272a6e4f1a492ef31a6b28300dbda9ee477927"} Jul 31 14:52:39.881021 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:39.880976 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c686965b-78r6c" podStartSLOduration=1.8809604640000002 podStartE2EDuration="1.880960464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:52:39.880676351 +0000 UTC m=+271.482112708" watchObservedRunningTime="2026-07-31 14:52:39.880960464 +0000 UTC m=+271.482396801" Jul 31 14:52:48.239117 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:48.239083 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:48.342936 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:48.342907 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:48.747178 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:48.747143 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:48.747389 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:48.747228 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:48.751921 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:48.751902 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:48.896825 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:48.896797 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:52:48.950303 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:48.950269 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7584c7fb9f-db426"] Jul 31 14:52:49.056837 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:49.056726 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:49.158663 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:49.158629 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:49.215474 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:52:49.215441 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:08.708414 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:08.708393 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 31 14:53:13.974809 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:13.974768 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7584c7fb9f-db426" podUID="ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" containerName="console" containerID="cri-o://4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e" gracePeriod=15 Jul 31 14:53:14.211727 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.211705 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:53:14.260712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.260658 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-oauth-config\") pod \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " Jul 31 14:53:14.260712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.260699 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-service-ca\") pod \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " Jul 31 14:53:14.260906 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.260729 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-serving-cert\") pod \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " Jul 31 14:53:14.260906 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.260753 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-config\") pod \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " Jul 31 14:53:14.260906 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.260804 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-trusted-ca-bundle\") pod \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " Jul 31 14:53:14.260906 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.260833 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-oauth-serving-cert\") pod \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " Jul 31 14:53:14.260906 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.260864 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxbll\" (UniqueName: \"kubernetes.io/projected/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-kube-api-access-dxbll\") pod \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\" (UID: \"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d\") " Jul 31 14:53:14.261202 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.261176 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-service-ca" (OuterVolumeSpecName: "service-ca") pod "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" (UID: "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:53:14.261357 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.261293 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-config" (OuterVolumeSpecName: "console-config") pod "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" (UID: "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:53:14.261357 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.261312 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" (UID: "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:53:14.261357 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.261309 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" (UID: "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:53:14.261607 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.261450 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-trusted-ca-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:53:14.261607 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.261468 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-oauth-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:53:14.261607 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.261482 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:53:14.261607 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.261498 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:53:14.262817 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.262794 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" (UID: "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:53:14.262906 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.262827 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" (UID: "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:53:14.262906 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.262866 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-kube-api-access-dxbll" (OuterVolumeSpecName: "kube-api-access-dxbll") pod "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" (UID: "ec2dce68-d02f-49fc-b7f9-0b9f60a3123d"). InnerVolumeSpecName "kube-api-access-dxbll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:53:14.362478 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.362457 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-oauth-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:53:14.362478 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.362477 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-console-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:53:14.362609 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.362488 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxbll\" (UniqueName: \"kubernetes.io/projected/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d-kube-api-access-dxbll\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:53:14.991716 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.991684 2582 generic.go:356] "Generic (PLEG): container finished" podID="ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" containerID="4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e" exitCode=0 Jul 31 14:53:14.992086 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.991747 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7584c7fb9f-db426" Jul 31 14:53:14.992086 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.991782 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7584c7fb9f-db426" event={"ID":"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d","Type":"ContainerDied","Data":"4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e"} Jul 31 14:53:14.992086 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.991858 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7584c7fb9f-db426" event={"ID":"ec2dce68-d02f-49fc-b7f9-0b9f60a3123d","Type":"ContainerDied","Data":"7581f096b1685edf40c03f0f4a681895a0ce82f4fcc47030805fc11129e5962e"} Jul 31 14:53:14.992086 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.991880 2582 scope.go:117] "RemoveContainer" containerID="4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e" Jul 31 14:53:14.999608 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.999591 2582 scope.go:117] "RemoveContainer" containerID="4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e" Jul 31 14:53:14.999868 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:53:14.999841 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e\": container with ID starting with 4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e not found: ID does not exist" containerID="4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e" Jul 31 14:53:14.999966 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:14.999879 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e"} err="failed to get container status \"4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e\": rpc error: code = NotFound desc = could not find container \"4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e\": container with ID starting with 4a4106b1ba2a7f732468edd1381e809401b9f9b73328e5a9fb68de990bc8f76e not found: ID does not exist" Jul 31 14:53:15.008715 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:15.008688 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7584c7fb9f-db426"] Jul 31 14:53:15.010167 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:15.010147 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7584c7fb9f-db426"] Jul 31 14:53:16.774507 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:16.774479 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" path="/var/lib/kubelet/pods/ec2dce68-d02f-49fc-b7f9-0b9f60a3123d/volumes" Jul 31 14:53:24.644453 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:24.644421 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:25.667913 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.667884 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76"] Jul 31 14:53:25.668368 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.668317 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" containerName="console" Jul 31 14:53:25.668368 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.668336 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" containerName="console" Jul 31 14:53:25.668488 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.668450 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2dce68-d02f-49fc-b7f9-0b9f60a3123d" containerName="console" Jul 31 14:53:25.672860 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.672839 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:25.675484 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.675459 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 14:53:25.675619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.675546 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b4497\"" Jul 31 14:53:25.675619 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.675611 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 14:53:25.680539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.680518 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76"] Jul 31 14:53:25.747206 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.747186 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsb6m\" (UniqueName: \"kubernetes.io/projected/3070b268-bb7f-478a-96b7-b26127398392-kube-api-access-fsb6m\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:25.747326 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.747215 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:25.747326 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.747242 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:25.783415 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.783385 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:25.847576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.847538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsb6m\" (UniqueName: \"kubernetes.io/projected/3070b268-bb7f-478a-96b7-b26127398392-kube-api-access-fsb6m\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:25.847669 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.847592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:25.847669 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.847636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:25.847949 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.847932 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:25.847995 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.847962 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:25.856494 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.856477 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsb6m\" (UniqueName: \"kubernetes.io/projected/3070b268-bb7f-478a-96b7-b26127398392-kube-api-access-fsb6m\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:25.982154 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:25.982087 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:26.114114 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:26.114059 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76"] Jul 31 14:53:26.116281 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:53:26.116250 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3070b268_bb7f_478a_96b7_b26127398392.slice/crio-51efad3de2afd82184bc6c1accc2de92b8c0665e4a545b00f51b6919311fa550 WatchSource:0}: Error finding container 51efad3de2afd82184bc6c1accc2de92b8c0665e4a545b00f51b6919311fa550: Status 404 returned error can't find the container with id 51efad3de2afd82184bc6c1accc2de92b8c0665e4a545b00f51b6919311fa550 Jul 31 14:53:26.118166 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:26.118148 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:53:27.038472 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:27.038423 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" event={"ID":"3070b268-bb7f-478a-96b7-b26127398392","Type":"ContainerStarted","Data":"51efad3de2afd82184bc6c1accc2de92b8c0665e4a545b00f51b6919311fa550"} Jul 31 14:53:33.065930 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:33.065896 2582 generic.go:356] "Generic (PLEG): container finished" podID="3070b268-bb7f-478a-96b7-b26127398392" containerID="7dfdc0719c8395796d8ba34c117d0b1f685685fe1723a66ef3ff9d43eb0b467a" exitCode=0 Jul 31 14:53:33.066295 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:33.065977 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" event={"ID":"3070b268-bb7f-478a-96b7-b26127398392","Type":"ContainerDied","Data":"7dfdc0719c8395796d8ba34c117d0b1f685685fe1723a66ef3ff9d43eb0b467a"} Jul 31 14:53:36.081768 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:36.081737 2582 generic.go:356] "Generic (PLEG): container finished" podID="3070b268-bb7f-478a-96b7-b26127398392" containerID="d06cdc8bcfa8a219772dbe88f17089bcd12712f029af56b8d7bd75f42ad667bb" exitCode=0 Jul 31 14:53:36.082195 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:36.081820 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" event={"ID":"3070b268-bb7f-478a-96b7-b26127398392","Type":"ContainerDied","Data":"d06cdc8bcfa8a219772dbe88f17089bcd12712f029af56b8d7bd75f42ad667bb"} Jul 31 14:53:43.117390 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:43.117356 2582 generic.go:356] "Generic (PLEG): container finished" podID="3070b268-bb7f-478a-96b7-b26127398392" containerID="1ad2d85ee865577d3756b2ffc4843adeefbf747ec444d78c241ca6b0a8b233fd" exitCode=0 Jul 31 14:53:43.117820 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:43.117418 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" event={"ID":"3070b268-bb7f-478a-96b7-b26127398392","Type":"ContainerDied","Data":"1ad2d85ee865577d3756b2ffc4843adeefbf747ec444d78c241ca6b0a8b233fd"} Jul 31 14:53:44.243787 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:44.243758 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:44.403606 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:44.403552 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-bundle\") pod \"3070b268-bb7f-478a-96b7-b26127398392\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " Jul 31 14:53:44.403793 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:44.403618 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsb6m\" (UniqueName: \"kubernetes.io/projected/3070b268-bb7f-478a-96b7-b26127398392-kube-api-access-fsb6m\") pod \"3070b268-bb7f-478a-96b7-b26127398392\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " Jul 31 14:53:44.403793 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:44.403678 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-util\") pod \"3070b268-bb7f-478a-96b7-b26127398392\" (UID: \"3070b268-bb7f-478a-96b7-b26127398392\") " Jul 31 14:53:44.404243 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:44.404218 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-bundle" (OuterVolumeSpecName: "bundle") pod "3070b268-bb7f-478a-96b7-b26127398392" (UID: "3070b268-bb7f-478a-96b7-b26127398392"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:53:44.405855 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:44.405832 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3070b268-bb7f-478a-96b7-b26127398392-kube-api-access-fsb6m" (OuterVolumeSpecName: "kube-api-access-fsb6m") pod "3070b268-bb7f-478a-96b7-b26127398392" (UID: "3070b268-bb7f-478a-96b7-b26127398392"). InnerVolumeSpecName "kube-api-access-fsb6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:53:44.408064 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:44.408043 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-util" (OuterVolumeSpecName: "util") pod "3070b268-bb7f-478a-96b7-b26127398392" (UID: "3070b268-bb7f-478a-96b7-b26127398392"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:53:44.504577 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:44.504536 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-util\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:53:44.504577 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:44.504569 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3070b268-bb7f-478a-96b7-b26127398392-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:53:44.504577 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:44.504578 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsb6m\" (UniqueName: \"kubernetes.io/projected/3070b268-bb7f-478a-96b7-b26127398392-kube-api-access-fsb6m\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:53:45.126772 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:45.126693 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" Jul 31 14:53:45.126911 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:45.126695 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b598t76" event={"ID":"3070b268-bb7f-478a-96b7-b26127398392","Type":"ContainerDied","Data":"51efad3de2afd82184bc6c1accc2de92b8c0665e4a545b00f51b6919311fa550"} Jul 31 14:53:45.126911 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:45.126812 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51efad3de2afd82184bc6c1accc2de92b8c0665e4a545b00f51b6919311fa550" Jul 31 14:53:47.233120 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:47.233090 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:47.359536 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:47.359506 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:48.789319 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.789293 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2"] Jul 31 14:53:48.789715 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.789582 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3070b268-bb7f-478a-96b7-b26127398392" containerName="util" Jul 31 14:53:48.789715 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.789596 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3070b268-bb7f-478a-96b7-b26127398392" containerName="util" Jul 31 14:53:48.789715 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.789605 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3070b268-bb7f-478a-96b7-b26127398392" containerName="pull" Jul 31 14:53:48.789715 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.789611 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3070b268-bb7f-478a-96b7-b26127398392" containerName="pull" Jul 31 14:53:48.789715 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.789623 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3070b268-bb7f-478a-96b7-b26127398392" containerName="extract" Jul 31 14:53:48.789715 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.789628 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3070b268-bb7f-478a-96b7-b26127398392" containerName="extract" Jul 31 14:53:48.789715 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.789683 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3070b268-bb7f-478a-96b7-b26127398392" containerName="extract" Jul 31 14:53:48.792953 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.792931 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" Jul 31 14:53:48.799155 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.798841 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 31 14:53:48.799155 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.799148 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-npdvq\"" Jul 31 14:53:48.799412 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.799398 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 31 14:53:48.799465 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.799411 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 31 14:53:48.811515 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.811491 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2"] Jul 31 14:53:48.944841 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.944814 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhnz\" (UniqueName: \"kubernetes.io/projected/ab51e30c-3bf1-433b-8cfd-d1ac38920111-kube-api-access-5vhnz\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-ltms2\" (UID: \"ab51e30c-3bf1-433b-8cfd-d1ac38920111\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" Jul 31 14:53:48.944993 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:48.944870 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ab51e30c-3bf1-433b-8cfd-d1ac38920111-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-ltms2\" (UID: \"ab51e30c-3bf1-433b-8cfd-d1ac38920111\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" Jul 31 14:53:49.045658 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:49.045584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ab51e30c-3bf1-433b-8cfd-d1ac38920111-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-ltms2\" (UID: \"ab51e30c-3bf1-433b-8cfd-d1ac38920111\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" Jul 31 14:53:49.045658 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:49.045652 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhnz\" (UniqueName: \"kubernetes.io/projected/ab51e30c-3bf1-433b-8cfd-d1ac38920111-kube-api-access-5vhnz\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-ltms2\" (UID: \"ab51e30c-3bf1-433b-8cfd-d1ac38920111\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" Jul 31 14:53:49.047796 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:49.047775 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ab51e30c-3bf1-433b-8cfd-d1ac38920111-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-ltms2\" (UID: \"ab51e30c-3bf1-433b-8cfd-d1ac38920111\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" Jul 31 14:53:49.062060 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:49.062041 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhnz\" (UniqueName: \"kubernetes.io/projected/ab51e30c-3bf1-433b-8cfd-d1ac38920111-kube-api-access-5vhnz\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-ltms2\" (UID: \"ab51e30c-3bf1-433b-8cfd-d1ac38920111\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" Jul 31 14:53:49.102330 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:49.102306 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" Jul 31 14:53:49.266461 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:49.266427 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2"] Jul 31 14:53:50.147360 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:50.147324 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" event={"ID":"ab51e30c-3bf1-433b-8cfd-d1ac38920111","Type":"ContainerStarted","Data":"d220393a622c065a62fb2faa8abb41ee09034440e5930719dff0c35486ed88ed"} Jul 31 14:53:53.161819 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.161789 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" event={"ID":"ab51e30c-3bf1-433b-8cfd-d1ac38920111","Type":"ContainerStarted","Data":"b5ba73498ed277f4b15b136bcb95186c406bf18f677b87e9a724d248068c2d6e"} Jul 31 14:53:53.162164 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.161860 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" Jul 31 14:53:53.181780 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.181755 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:53.182398 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.182363 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" podStartSLOduration=1.78534383 podStartE2EDuration="5.182348853s" totalImagesPullingTime="3.397005023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:53:53.179958256 +0000 UTC m=+344.781394594" watchObservedRunningTime="2026-07-31 14:53:53.182348853 +0000 UTC m=+344.783785195" Jul 31 14:53:53.524500 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.524419 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:53.536107 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.536083 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-cxp5d"] Jul 31 14:53:53.545097 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.545078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:53.547430 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.547404 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 31 14:53:53.547539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.547483 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 31 14:53:53.547689 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.547674 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zpntd\"" Jul 31 14:53:53.550649 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.550630 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-cxp5d"] Jul 31 14:53:53.621222 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.621185 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:53.684167 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.684142 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/303f757c-80d4-4d2e-b64f-3daa109109b0-cabundle0\") pod \"keda-operator-77f648d694-cxp5d\" (UID: \"303f757c-80d4-4d2e-b64f-3daa109109b0\") " pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:53.684266 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.684205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r6dz\" (UniqueName: \"kubernetes.io/projected/303f757c-80d4-4d2e-b64f-3daa109109b0-kube-api-access-4r6dz\") pod \"keda-operator-77f648d694-cxp5d\" (UID: \"303f757c-80d4-4d2e-b64f-3daa109109b0\") " pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:53.684325 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.684274 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/303f757c-80d4-4d2e-b64f-3daa109109b0-certificates\") pod \"keda-operator-77f648d694-cxp5d\" (UID: \"303f757c-80d4-4d2e-b64f-3daa109109b0\") " pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:53.724084 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.724048 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:53.785021 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.784961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/303f757c-80d4-4d2e-b64f-3daa109109b0-certificates\") pod \"keda-operator-77f648d694-cxp5d\" (UID: \"303f757c-80d4-4d2e-b64f-3daa109109b0\") " pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:53.785021 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.785018 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/303f757c-80d4-4d2e-b64f-3daa109109b0-cabundle0\") pod \"keda-operator-77f648d694-cxp5d\" (UID: \"303f757c-80d4-4d2e-b64f-3daa109109b0\") " pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:53.785162 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.785064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6dz\" (UniqueName: \"kubernetes.io/projected/303f757c-80d4-4d2e-b64f-3daa109109b0-kube-api-access-4r6dz\") pod \"keda-operator-77f648d694-cxp5d\" (UID: \"303f757c-80d4-4d2e-b64f-3daa109109b0\") " pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:53.785162 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:53:53.785101 2582 secret.go:281] references non-existent secret key: ca.crt Jul 31 14:53:53.785162 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:53:53.785119 2582 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 31 14:53:53.785162 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:53:53.785128 2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-cxp5d: references non-existent secret key: ca.crt Jul 31 14:53:53.785315 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:53:53.785176 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/303f757c-80d4-4d2e-b64f-3daa109109b0-certificates podName:303f757c-80d4-4d2e-b64f-3daa109109b0 nodeName:}" failed. No retries permitted until 2026-07-31 14:53:54.285160006 +0000 UTC m=+345.886596322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/303f757c-80d4-4d2e-b64f-3daa109109b0-certificates") pod "keda-operator-77f648d694-cxp5d" (UID: "303f757c-80d4-4d2e-b64f-3daa109109b0") : references non-existent secret key: ca.crt Jul 31 14:53:53.785710 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.785694 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/303f757c-80d4-4d2e-b64f-3daa109109b0-cabundle0\") pod \"keda-operator-77f648d694-cxp5d\" (UID: \"303f757c-80d4-4d2e-b64f-3daa109109b0\") " pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:53.793352 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.793335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r6dz\" (UniqueName: \"kubernetes.io/projected/303f757c-80d4-4d2e-b64f-3daa109109b0-kube-api-access-4r6dz\") pod \"keda-operator-77f648d694-cxp5d\" (UID: \"303f757c-80d4-4d2e-b64f-3daa109109b0\") " pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:53.960722 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:53.960691 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:54.126777 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.126682 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:54.126950 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.126930 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-xrfsx"] Jul 31 14:53:54.130311 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.130293 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" Jul 31 14:53:54.132410 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.132392 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 31 14:53:54.139751 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.139728 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-xrfsx"] Jul 31 14:53:54.224084 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.224057 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:54.288253 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.288231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11-certificates\") pod \"keda-admission-56f9798bb4-xrfsx\" (UID: \"8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11\") " pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" Jul 31 14:53:54.288360 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.288266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/303f757c-80d4-4d2e-b64f-3daa109109b0-certificates\") pod \"keda-operator-77f648d694-cxp5d\" (UID: \"303f757c-80d4-4d2e-b64f-3daa109109b0\") " pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:54.288360 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.288352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bd84\" (UniqueName: \"kubernetes.io/projected/8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11-kube-api-access-7bd84\") pod \"keda-admission-56f9798bb4-xrfsx\" (UID: \"8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11\") " pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" Jul 31 14:53:54.290634 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.290615 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/303f757c-80d4-4d2e-b64f-3daa109109b0-certificates\") pod \"keda-operator-77f648d694-cxp5d\" (UID: \"303f757c-80d4-4d2e-b64f-3daa109109b0\") " pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:54.326811 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.326788 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:54.388982 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.388956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bd84\" (UniqueName: \"kubernetes.io/projected/8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11-kube-api-access-7bd84\") pod \"keda-admission-56f9798bb4-xrfsx\" (UID: \"8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11\") " pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" Jul 31 14:53:54.389090 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.389033 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11-certificates\") pod \"keda-admission-56f9798bb4-xrfsx\" (UID: \"8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11\") " pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" Jul 31 14:53:54.391251 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.391234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11-certificates\") pod \"keda-admission-56f9798bb4-xrfsx\" (UID: \"8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11\") " pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" Jul 31 14:53:54.400978 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.400959 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bd84\" (UniqueName: \"kubernetes.io/projected/8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11-kube-api-access-7bd84\") pod \"keda-admission-56f9798bb4-xrfsx\" (UID: \"8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11\") " pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" Jul 31 14:53:54.427610 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.427589 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:54.441727 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.441706 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" Jul 31 14:53:54.456666 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.456638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:53:54.616551 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.616524 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-xrfsx"] Jul 31 14:53:54.619408 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:53:54.619382 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddb5b38_7e3f_4c5b_b8e4_14a669adcb11.slice/crio-6d874810b7c8bc3574260c504d71f0557ac0fa62e329e4c0e04bb39bba7f59d1 WatchSource:0}: Error finding container 6d874810b7c8bc3574260c504d71f0557ac0fa62e329e4c0e04bb39bba7f59d1: Status 404 returned error can't find the container with id 6d874810b7c8bc3574260c504d71f0557ac0fa62e329e4c0e04bb39bba7f59d1 Jul 31 14:53:54.622482 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.622463 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-cxp5d"] Jul 31 14:53:54.624671 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:53:54.624648 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303f757c_80d4_4d2e_b64f_3daa109109b0.slice/crio-e66cb74a4d81a236b6198d75d83207920521b973c39f23d947620bdb610f7b3f WatchSource:0}: Error finding container e66cb74a4d81a236b6198d75d83207920521b973c39f23d947620bdb610f7b3f: Status 404 returned error can't find the container with id e66cb74a4d81a236b6198d75d83207920521b973c39f23d947620bdb610f7b3f Jul 31 14:53:54.813592 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:54.813508 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:55.000944 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:55.000915 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:55.171850 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:55.171818 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-cxp5d" event={"ID":"303f757c-80d4-4d2e-b64f-3daa109109b0","Type":"ContainerStarted","Data":"e66cb74a4d81a236b6198d75d83207920521b973c39f23d947620bdb610f7b3f"} Jul 31 14:53:55.172787 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:55.172763 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" event={"ID":"8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11","Type":"ContainerStarted","Data":"6d874810b7c8bc3574260c504d71f0557ac0fa62e329e4c0e04bb39bba7f59d1"} Jul 31 14:53:55.185112 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:55.185087 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:55.312497 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:55.312464 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:57.183797 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:57.183766 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" event={"ID":"8ddb5b38-7e3f-4c5b-b8e4-14a669adcb11","Type":"ContainerStarted","Data":"95d5e72aef5bca4284415805589babf1c44bfaabac39e2282f232a72b71bdd06"} Jul 31 14:53:57.184171 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:57.183883 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" Jul 31 14:53:57.206800 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:53:57.206756 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" podStartSLOduration=1.1941075780000001 podStartE2EDuration="3.2067434s" totalImagesPullingTime="2.012635822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:53:57.204355392 +0000 UTC m=+348.805791725" watchObservedRunningTime="2026-07-31 14:53:57.2067434 +0000 UTC m=+348.808179737" Jul 31 14:54:07.226800 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:07.226769 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-cxp5d" event={"ID":"303f757c-80d4-4d2e-b64f-3daa109109b0","Type":"ContainerStarted","Data":"b563cd6509a7cebc3ad500826bb47b5170ee52d7694359ae6537350da3cfa914"} Jul 31 14:54:07.227190 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:07.226902 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:54:14.168997 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:14.168967 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-ltms2" Jul 31 14:54:14.196542 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:14.196490 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-cxp5d" podStartSLOduration=9.227997599 podStartE2EDuration="21.196472694s" totalImagesPullingTime="11.968475095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:54:07.248632287 +0000 UTC m=+358.850068622" watchObservedRunningTime="2026-07-31 14:54:14.196472694 +0000 UTC m=+365.797909029" Jul 31 14:54:18.190716 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:18.190683 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-xrfsx" Jul 31 14:54:28.234505 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:28.234471 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-cxp5d" Jul 31 14:54:46.652829 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:46.652792 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:46.764983 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:46.764917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:47.108431 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.108345 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn"] Jul 31 14:54:47.114907 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.114885 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.117312 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.117292 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 14:54:47.117500 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.117486 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b4497\"" Jul 31 14:54:47.118084 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.118070 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 14:54:47.134016 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.133990 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn"] Jul 31 14:54:47.195645 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.195607 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.195747 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.195670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.195747 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.195715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj876\" (UniqueName: \"kubernetes.io/projected/1fe6747f-36ec-4a0d-b729-2156e07f598f-kube-api-access-xj876\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.296317 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.296284 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj876\" (UniqueName: \"kubernetes.io/projected/1fe6747f-36ec-4a0d-b729-2156e07f598f-kube-api-access-xj876\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.296422 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.296342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.296422 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.296418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.296706 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.296690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.296753 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.296743 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.308450 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.308428 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj876\" (UniqueName: \"kubernetes.io/projected/1fe6747f-36ec-4a0d-b729-2156e07f598f-kube-api-access-xj876\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.423378 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.423356 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:47.572348 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:47.572321 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn"] Jul 31 14:54:47.574343 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:54:47.574314 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe6747f_36ec_4a0d_b729_2156e07f598f.slice/crio-589e96620da5991ef4bfaa593b909a81d299a9452da86446ec797dd2ba325e66 WatchSource:0}: Error finding container 589e96620da5991ef4bfaa593b909a81d299a9452da86446ec797dd2ba325e66: Status 404 returned error can't find the container with id 589e96620da5991ef4bfaa593b909a81d299a9452da86446ec797dd2ba325e66 Jul 31 14:54:48.399825 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:48.399799 2582 generic.go:356] "Generic (PLEG): container finished" podID="1fe6747f-36ec-4a0d-b729-2156e07f598f" containerID="cf1cde3666c9346ddea135c194a650bcf67e6e9969327c2520666ffb3d66f466" exitCode=0 Jul 31 14:54:48.400198 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:48.399888 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" event={"ID":"1fe6747f-36ec-4a0d-b729-2156e07f598f","Type":"ContainerDied","Data":"cf1cde3666c9346ddea135c194a650bcf67e6e9969327c2520666ffb3d66f466"} Jul 31 14:54:48.400198 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:48.399937 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" event={"ID":"1fe6747f-36ec-4a0d-b729-2156e07f598f","Type":"ContainerStarted","Data":"589e96620da5991ef4bfaa593b909a81d299a9452da86446ec797dd2ba325e66"} Jul 31 14:54:50.411540 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:50.411501 2582 generic.go:356] "Generic (PLEG): container finished" podID="1fe6747f-36ec-4a0d-b729-2156e07f598f" containerID="59e6d2180ce2f814b08e7ee9e39b3b531448dd2d08692937263ca3036c4dca98" exitCode=0 Jul 31 14:54:50.411852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:50.411574 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" event={"ID":"1fe6747f-36ec-4a0d-b729-2156e07f598f","Type":"ContainerDied","Data":"59e6d2180ce2f814b08e7ee9e39b3b531448dd2d08692937263ca3036c4dca98"} Jul 31 14:54:51.420227 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:51.420195 2582 generic.go:356] "Generic (PLEG): container finished" podID="1fe6747f-36ec-4a0d-b729-2156e07f598f" containerID="9226516c9efcf0677e9ac42038455ea6ff9f49a7cd0e916e5af29ccdbc610881" exitCode=0 Jul 31 14:54:51.420622 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:51.420284 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" event={"ID":"1fe6747f-36ec-4a0d-b729-2156e07f598f","Type":"ContainerDied","Data":"9226516c9efcf0677e9ac42038455ea6ff9f49a7cd0e916e5af29ccdbc610881"} Jul 31 14:54:52.542874 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:52.542854 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:52.640418 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:52.640389 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-bundle\") pod \"1fe6747f-36ec-4a0d-b729-2156e07f598f\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " Jul 31 14:54:52.640418 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:52.640420 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj876\" (UniqueName: \"kubernetes.io/projected/1fe6747f-36ec-4a0d-b729-2156e07f598f-kube-api-access-xj876\") pod \"1fe6747f-36ec-4a0d-b729-2156e07f598f\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " Jul 31 14:54:52.640592 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:52.640450 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-util\") pod \"1fe6747f-36ec-4a0d-b729-2156e07f598f\" (UID: \"1fe6747f-36ec-4a0d-b729-2156e07f598f\") " Jul 31 14:54:52.641148 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:52.641119 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-bundle" (OuterVolumeSpecName: "bundle") pod "1fe6747f-36ec-4a0d-b729-2156e07f598f" (UID: "1fe6747f-36ec-4a0d-b729-2156e07f598f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:54:52.642406 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:52.642387 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe6747f-36ec-4a0d-b729-2156e07f598f-kube-api-access-xj876" (OuterVolumeSpecName: "kube-api-access-xj876") pod "1fe6747f-36ec-4a0d-b729-2156e07f598f" (UID: "1fe6747f-36ec-4a0d-b729-2156e07f598f"). InnerVolumeSpecName "kube-api-access-xj876". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:54:52.645816 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:52.645783 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-util" (OuterVolumeSpecName: "util") pod "1fe6747f-36ec-4a0d-b729-2156e07f598f" (UID: "1fe6747f-36ec-4a0d-b729-2156e07f598f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:54:52.741581 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:52.741508 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:54:52.741581 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:52.741529 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj876\" (UniqueName: \"kubernetes.io/projected/1fe6747f-36ec-4a0d-b729-2156e07f598f-kube-api-access-xj876\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:54:52.741581 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:52.741539 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fe6747f-36ec-4a0d-b729-2156e07f598f-util\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:54:53.430006 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:53.429976 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" Jul 31 14:54:53.430174 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:53.429999 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9nbn" event={"ID":"1fe6747f-36ec-4a0d-b729-2156e07f598f","Type":"ContainerDied","Data":"589e96620da5991ef4bfaa593b909a81d299a9452da86446ec797dd2ba325e66"} Jul 31 14:54:53.430174 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:53.430050 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="589e96620da5991ef4bfaa593b909a81d299a9452da86446ec797dd2ba325e66" Jul 31 14:54:57.984571 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:57.984522 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:58.674505 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:54:58.674476 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:00.029096 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.029067 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c"] Jul 31 14:55:00.029439 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.029390 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe6747f-36ec-4a0d-b729-2156e07f598f" containerName="util" Jul 31 14:55:00.029439 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.029404 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe6747f-36ec-4a0d-b729-2156e07f598f" containerName="util" Jul 31 14:55:00.029439 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.029421 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe6747f-36ec-4a0d-b729-2156e07f598f" containerName="extract" Jul 31 14:55:00.029439 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.029426 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe6747f-36ec-4a0d-b729-2156e07f598f" containerName="extract" Jul 31 14:55:00.029439 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.029436 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe6747f-36ec-4a0d-b729-2156e07f598f" containerName="pull" Jul 31 14:55:00.029439 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.029442 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe6747f-36ec-4a0d-b729-2156e07f598f" containerName="pull" Jul 31 14:55:00.029668 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.029493 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe6747f-36ec-4a0d-b729-2156e07f598f" containerName="extract" Jul 31 14:55:00.032633 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.032619 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" Jul 31 14:55:00.036389 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.036368 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:55:00.036495 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.036375 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wltnm\"" Jul 31 14:55:00.039697 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.039676 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 31 14:55:00.050945 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.050920 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c"] Jul 31 14:55:00.202799 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.202780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ljq\" (UniqueName: \"kubernetes.io/projected/18fef3b5-037e-4422-beb3-2be500039c97-kube-api-access-99ljq\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tpb6c\" (UID: \"18fef3b5-037e-4422-beb3-2be500039c97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" Jul 31 14:55:00.202895 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.202848 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18fef3b5-037e-4422-beb3-2be500039c97-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tpb6c\" (UID: \"18fef3b5-037e-4422-beb3-2be500039c97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" Jul 31 14:55:00.303618 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.303543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18fef3b5-037e-4422-beb3-2be500039c97-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tpb6c\" (UID: \"18fef3b5-037e-4422-beb3-2be500039c97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" Jul 31 14:55:00.303712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.303620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ljq\" (UniqueName: \"kubernetes.io/projected/18fef3b5-037e-4422-beb3-2be500039c97-kube-api-access-99ljq\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tpb6c\" (UID: \"18fef3b5-037e-4422-beb3-2be500039c97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" Jul 31 14:55:00.303884 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.303866 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18fef3b5-037e-4422-beb3-2be500039c97-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tpb6c\" (UID: \"18fef3b5-037e-4422-beb3-2be500039c97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" Jul 31 14:55:00.312326 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.312300 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ljq\" (UniqueName: \"kubernetes.io/projected/18fef3b5-037e-4422-beb3-2be500039c97-kube-api-access-99ljq\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tpb6c\" (UID: \"18fef3b5-037e-4422-beb3-2be500039c97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" Jul 31 14:55:00.340932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.340914 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" Jul 31 14:55:00.472277 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:00.472231 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c"] Jul 31 14:55:00.474464 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:55:00.474439 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fef3b5_037e_4422_beb3_2be500039c97.slice/crio-eb450c1f41c320ab8cb99682671d7a1c05f36932c2168caa32279c815b60a804 WatchSource:0}: Error finding container eb450c1f41c320ab8cb99682671d7a1c05f36932c2168caa32279c815b60a804: Status 404 returned error can't find the container with id eb450c1f41c320ab8cb99682671d7a1c05f36932c2168caa32279c815b60a804 Jul 31 14:55:01.465940 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:01.465909 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" event={"ID":"18fef3b5-037e-4422-beb3-2be500039c97","Type":"ContainerStarted","Data":"eb450c1f41c320ab8cb99682671d7a1c05f36932c2168caa32279c815b60a804"} Jul 31 14:55:01.778780 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:01.778711 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:01.899432 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:01.899404 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:04.441695 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:04.441663 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:04.482114 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:04.482082 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" event={"ID":"18fef3b5-037e-4422-beb3-2be500039c97","Type":"ContainerStarted","Data":"80a5dca877501a22db09d37ad7f450ea4181a79c30fe6e332c999cd919f50607"} Jul 31 14:55:04.511396 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:04.511352 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tpb6c" podStartSLOduration=2.271145471 podStartE2EDuration="5.511336522s" totalImagesPullingTime="3.240191051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:04.50766766 +0000 UTC m=+416.109103999" watchObservedRunningTime="2026-07-31 14:55:04.511336522 +0000 UTC m=+416.112772858" Jul 31 14:55:04.544330 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:04.544124 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:06.584692 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:06.584663 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:06.719180 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:06.719142 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:07.130178 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:07.130149 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:08.209838 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:08.209807 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:08.356024 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:08.355996 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:08.456580 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:08.456519 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:09.307183 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.307147 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s"] Jul 31 14:55:09.310598 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.310581 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.313152 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.313114 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 14:55:09.313268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.313249 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b4497\"" Jul 31 14:55:09.314227 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.314209 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 14:55:09.322014 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.321996 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s"] Jul 31 14:55:09.377440 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.377416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.377584 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.377461 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.377584 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.377530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f86cq\" (UniqueName: \"kubernetes.io/projected/c434a926-5827-472f-a863-af5d8bd51f24-kube-api-access-f86cq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.477920 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.477883 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.478107 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.477932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.478107 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.477975 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f86cq\" (UniqueName: \"kubernetes.io/projected/c434a926-5827-472f-a863-af5d8bd51f24-kube-api-access-f86cq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.478270 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.478250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.478333 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.478278 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.495365 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.495335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f86cq\" (UniqueName: \"kubernetes.io/projected/c434a926-5827-472f-a863-af5d8bd51f24-kube-api-access-f86cq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.620092 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.620024 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:09.753891 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:09.753828 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s"] Jul 31 14:55:10.510413 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:10.510384 2582 generic.go:356] "Generic (PLEG): container finished" podID="c434a926-5827-472f-a863-af5d8bd51f24" containerID="f145db27494f1f41e196ab32f91697469015a3c1de63b18830dbb28661b8cc7f" exitCode=0 Jul 31 14:55:10.510797 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:10.510428 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" event={"ID":"c434a926-5827-472f-a863-af5d8bd51f24","Type":"ContainerDied","Data":"f145db27494f1f41e196ab32f91697469015a3c1de63b18830dbb28661b8cc7f"} Jul 31 14:55:10.510797 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:10.510463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" event={"ID":"c434a926-5827-472f-a863-af5d8bd51f24","Type":"ContainerStarted","Data":"a23df123d35580843268adfc40de0c09f4f5744cac52995688a8ab188ce0eb72"} Jul 31 14:55:13.528307 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:13.528270 2582 generic.go:356] "Generic (PLEG): container finished" podID="c434a926-5827-472f-a863-af5d8bd51f24" containerID="18638a262b279b9fbd16b3b715c8218857d4b688f7627cb35eb90c97f43c1fb4" exitCode=0 Jul 31 14:55:13.528683 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:13.528315 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" event={"ID":"c434a926-5827-472f-a863-af5d8bd51f24","Type":"ContainerDied","Data":"18638a262b279b9fbd16b3b715c8218857d4b688f7627cb35eb90c97f43c1fb4"} Jul 31 14:55:14.535507 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:14.535474 2582 generic.go:356] "Generic (PLEG): container finished" podID="c434a926-5827-472f-a863-af5d8bd51f24" containerID="c02318868a9c4c6b0a0557b9018a5d974017863f5c7fbf60883aaf1c80e60b1f" exitCode=0 Jul 31 14:55:14.535901 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:14.535519 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" event={"ID":"c434a926-5827-472f-a863-af5d8bd51f24","Type":"ContainerDied","Data":"c02318868a9c4c6b0a0557b9018a5d974017863f5c7fbf60883aaf1c80e60b1f"} Jul 31 14:55:15.221757 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.221723 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:15.671205 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.671179 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:15.725920 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.725891 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-bundle\") pod \"c434a926-5827-472f-a863-af5d8bd51f24\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " Jul 31 14:55:15.726058 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.725962 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-util\") pod \"c434a926-5827-472f-a863-af5d8bd51f24\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " Jul 31 14:55:15.726058 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.726011 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f86cq\" (UniqueName: \"kubernetes.io/projected/c434a926-5827-472f-a863-af5d8bd51f24-kube-api-access-f86cq\") pod \"c434a926-5827-472f-a863-af5d8bd51f24\" (UID: \"c434a926-5827-472f-a863-af5d8bd51f24\") " Jul 31 14:55:15.726332 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.726301 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-bundle" (OuterVolumeSpecName: "bundle") pod "c434a926-5827-472f-a863-af5d8bd51f24" (UID: "c434a926-5827-472f-a863-af5d8bd51f24"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:55:15.728053 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.728024 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c434a926-5827-472f-a863-af5d8bd51f24-kube-api-access-f86cq" (OuterVolumeSpecName: "kube-api-access-f86cq") pod "c434a926-5827-472f-a863-af5d8bd51f24" (UID: "c434a926-5827-472f-a863-af5d8bd51f24"). InnerVolumeSpecName "kube-api-access-f86cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:55:15.730277 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.730244 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-util" (OuterVolumeSpecName: "util") pod "c434a926-5827-472f-a863-af5d8bd51f24" (UID: "c434a926-5827-472f-a863-af5d8bd51f24"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:55:15.827493 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.827423 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:55:15.827493 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.827446 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c434a926-5827-472f-a863-af5d8bd51f24-util\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:55:15.827493 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:15.827454 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f86cq\" (UniqueName: \"kubernetes.io/projected/c434a926-5827-472f-a863-af5d8bd51f24-kube-api-access-f86cq\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:55:16.546233 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:16.546197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" event={"ID":"c434a926-5827-472f-a863-af5d8bd51f24","Type":"ContainerDied","Data":"a23df123d35580843268adfc40de0c09f4f5744cac52995688a8ab188ce0eb72"} Jul 31 14:55:16.546233 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:16.546225 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f82j7s" Jul 31 14:55:16.546427 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:16.546246 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a23df123d35580843268adfc40de0c09f4f5744cac52995688a8ab188ce0eb72" Jul 31 14:55:17.476032 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.475998 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ffd95fdd5-k42db"] Jul 31 14:55:17.476452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.476304 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c434a926-5827-472f-a863-af5d8bd51f24" containerName="pull" Jul 31 14:55:17.476452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.476313 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c434a926-5827-472f-a863-af5d8bd51f24" containerName="pull" Jul 31 14:55:17.476452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.476334 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c434a926-5827-472f-a863-af5d8bd51f24" containerName="util" Jul 31 14:55:17.476452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.476341 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c434a926-5827-472f-a863-af5d8bd51f24" containerName="util" Jul 31 14:55:17.476452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.476348 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c434a926-5827-472f-a863-af5d8bd51f24" containerName="extract" Jul 31 14:55:17.476452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.476353 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c434a926-5827-472f-a863-af5d8bd51f24" containerName="extract" Jul 31 14:55:17.476452 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.476399 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c434a926-5827-472f-a863-af5d8bd51f24" containerName="extract" Jul 31 14:55:17.479199 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.479173 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.490181 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.490160 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ffd95fdd5-k42db"] Jul 31 14:55:17.540730 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.540705 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-oauth-serving-cert\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.540856 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.540743 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40296959-a9e9-4b82-a90c-dae187dcc097-console-serving-cert\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.540856 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.540759 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40296959-a9e9-4b82-a90c-dae187dcc097-console-oauth-config\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.540856 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.540822 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-service-ca\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.540856 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.540843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-trusted-ca-bundle\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.541028 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.540868 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-console-config\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.541028 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.540889 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpshj\" (UniqueName: \"kubernetes.io/projected/40296959-a9e9-4b82-a90c-dae187dcc097-kube-api-access-vpshj\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.547256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.547235 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:17.619070 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.619044 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:17.641844 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.641818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-oauth-serving-cert\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.641961 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.641859 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40296959-a9e9-4b82-a90c-dae187dcc097-console-serving-cert\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.641961 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.641884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40296959-a9e9-4b82-a90c-dae187dcc097-console-oauth-config\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.641961 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.641938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-service-ca\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.642124 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.641967 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-trusted-ca-bundle\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.642124 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.642003 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-console-config\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.642124 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.642038 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpshj\" (UniqueName: \"kubernetes.io/projected/40296959-a9e9-4b82-a90c-dae187dcc097-kube-api-access-vpshj\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.642714 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.642690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-oauth-serving-cert\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.642826 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.642769 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-service-ca\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.642893 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.642846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-console-config\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.642952 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.642915 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40296959-a9e9-4b82-a90c-dae187dcc097-trusted-ca-bundle\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.644338 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.644312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40296959-a9e9-4b82-a90c-dae187dcc097-console-serving-cert\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.644423 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.644340 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40296959-a9e9-4b82-a90c-dae187dcc097-console-oauth-config\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.651594 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.651537 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpshj\" (UniqueName: \"kubernetes.io/projected/40296959-a9e9-4b82-a90c-dae187dcc097-kube-api-access-vpshj\") pod \"console-ffd95fdd5-k42db\" (UID: \"40296959-a9e9-4b82-a90c-dae187dcc097\") " pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.699622 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.699598 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:17.788725 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.788666 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:17.922809 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:17.922711 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ffd95fdd5-k42db"] Jul 31 14:55:17.925069 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:55:17.925041 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40296959_a9e9_4b82_a90c_dae187dcc097.slice/crio-39f84702ccea8b82858d90851cade8719a72a70566f9fa4cb9f32186891eac7e WatchSource:0}: Error finding container 39f84702ccea8b82858d90851cade8719a72a70566f9fa4cb9f32186891eac7e: Status 404 returned error can't find the container with id 39f84702ccea8b82858d90851cade8719a72a70566f9fa4cb9f32186891eac7e Jul 31 14:55:18.279215 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:18.279184 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:18.386638 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:18.386604 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:18.555549 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:18.555471 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffd95fdd5-k42db" event={"ID":"40296959-a9e9-4b82-a90c-dae187dcc097","Type":"ContainerStarted","Data":"7463f3075229c19620171d3e4bd56cbac74914a9b5887e1edc091de863e24cd6"} Jul 31 14:55:18.555549 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:18.555520 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffd95fdd5-k42db" event={"ID":"40296959-a9e9-4b82-a90c-dae187dcc097","Type":"ContainerStarted","Data":"39f84702ccea8b82858d90851cade8719a72a70566f9fa4cb9f32186891eac7e"} Jul 31 14:55:18.575232 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:18.575182 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ffd95fdd5-k42db" podStartSLOduration=1.575160768 podStartE2EDuration="1.575160768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:18.57445822 +0000 UTC m=+430.175894557" watchObservedRunningTime="2026-07-31 14:55:18.575160768 +0000 UTC m=+430.176597158" Jul 31 14:55:21.644955 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:21.644922 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:21.743149 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:21.743115 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:27.789194 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:27.789156 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:27.789709 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:27.789221 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:27.793867 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:27.793841 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:28.606323 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:28.606289 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ffd95fdd5-k42db" Jul 31 14:55:28.661803 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:28.661763 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c686965b-78r6c"] Jul 31 14:55:28.776263 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:28.776234 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:28.875391 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:28.875310 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:28.976228 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:28.976183 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:31.979887 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:31.979805 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:37.375327 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.375296 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:37.386973 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.386942 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs"] Jul 31 14:55:37.390451 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.390430 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.393039 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.393020 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 31 14:55:37.393386 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.393368 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 31 14:55:37.393437 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.393403 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 31 14:55:37.393508 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.393491 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fg9l8\"" Jul 31 14:55:37.393732 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.393719 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 31 14:55:37.413259 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.413238 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs"] Jul 31 14:55:37.456137 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.456108 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:37.515343 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.515315 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.515468 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.515347 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.515468 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.515380 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.515468 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.515410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr4sw\" (UniqueName: \"kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-kube-api-access-xr4sw\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.515629 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.515478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.515629 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.515529 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.515629 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.515584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.616391 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.616355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.616592 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.616401 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.616592 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.616439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.616592 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.616472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.616592 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.616526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.616592 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.616583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4sw\" (UniqueName: \"kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-kube-api-access-xr4sw\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.616903 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.616616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.617205 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.617183 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.619278 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.619253 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.619427 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.619310 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.619517 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.619353 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.624859 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.618776 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.630656 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.630631 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4sw\" (UniqueName: \"kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-kube-api-access-xr4sw\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.632217 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.632199 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mw7hs\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.700943 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.700910 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:37.841822 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:37.841791 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs"] Jul 31 14:55:37.846980 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:55:37.846956 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd029c9_b3e4_41d2_bbec_9e11f4a7af68.slice/crio-4864e30a2fcb3001fc7d92add3eff3c11d69efd8693b480eb1a2fd749ddd978a WatchSource:0}: Error finding container 4864e30a2fcb3001fc7d92add3eff3c11d69efd8693b480eb1a2fd749ddd978a: Status 404 returned error can't find the container with id 4864e30a2fcb3001fc7d92add3eff3c11d69efd8693b480eb1a2fd749ddd978a Jul 31 14:55:38.647726 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:38.647694 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" event={"ID":"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68","Type":"ContainerStarted","Data":"4864e30a2fcb3001fc7d92add3eff3c11d69efd8693b480eb1a2fd749ddd978a"} Jul 31 14:55:38.862622 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:38.862596 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:40.760880 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:40.760849 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:40.826719 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:40.826688 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:40.993139 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:40.993105 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 14:55:40.993217 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:40.993199 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 14:55:41.541123 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:41.541088 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:41.604788 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:41.604753 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:41.666514 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:41.666479 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" event={"ID":"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68","Type":"ContainerStarted","Data":"740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623"} Jul 31 14:55:41.666701 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:41.666586 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:41.736766 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:41.736707 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" podStartSLOduration=1.592744292 podStartE2EDuration="4.736688573s" totalImagesPullingTime="3.143944281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:41.736093296 +0000 UTC m=+453.337529634" watchObservedRunningTime="2026-07-31 14:55:41.736688573 +0000 UTC m=+453.338124909" Jul 31 14:55:41.817326 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:41.817239 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:41.912846 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:41.912817 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:42.679130 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:42.679098 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-mw7hs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 14:55:42.679276 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:42.679160 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" podUID="5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 14:55:45.677990 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:45.677962 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:55:46.438027 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:46.437994 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:46.592121 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:46.592087 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:50.019949 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:50.019919 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:53.683650 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:53.683604 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c686965b-78r6c" podUID="ec38bdde-60ff-4195-8da2-1dd8b81e514c" containerName="console" containerID="cri-o://ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a" gracePeriod=15 Jul 31 14:55:53.930065 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:53.930040 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:55:53.937393 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:53.937288 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:54.041852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.041815 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:54.063672 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.063641 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-trusted-ca-bundle\") pod \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " Jul 31 14:55:54.063672 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.063677 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-service-ca\") pod \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " Jul 31 14:55:54.063909 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.063714 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-oauth-serving-cert\") pod \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " Jul 31 14:55:54.063909 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.063737 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-oauth-config\") pod \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " Jul 31 14:55:54.063909 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.063768 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p95r\" (UniqueName: \"kubernetes.io/projected/ec38bdde-60ff-4195-8da2-1dd8b81e514c-kube-api-access-9p95r\") pod \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " Jul 31 14:55:54.063909 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.063870 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-serving-cert\") pod \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " Jul 31 14:55:54.063909 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.063893 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-config\") pod \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\" (UID: \"ec38bdde-60ff-4195-8da2-1dd8b81e514c\") " Jul 31 14:55:54.064405 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.064126 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-service-ca" (OuterVolumeSpecName: "service-ca") pod "ec38bdde-60ff-4195-8da2-1dd8b81e514c" (UID: "ec38bdde-60ff-4195-8da2-1dd8b81e514c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:55:54.064405 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.064132 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ec38bdde-60ff-4195-8da2-1dd8b81e514c" (UID: "ec38bdde-60ff-4195-8da2-1dd8b81e514c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:55:54.064405 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.064284 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ec38bdde-60ff-4195-8da2-1dd8b81e514c" (UID: "ec38bdde-60ff-4195-8da2-1dd8b81e514c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:55:54.064405 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.064376 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-config" (OuterVolumeSpecName: "console-config") pod "ec38bdde-60ff-4195-8da2-1dd8b81e514c" (UID: "ec38bdde-60ff-4195-8da2-1dd8b81e514c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:55:54.066100 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.066080 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ec38bdde-60ff-4195-8da2-1dd8b81e514c" (UID: "ec38bdde-60ff-4195-8da2-1dd8b81e514c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:55:54.066457 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.066440 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec38bdde-60ff-4195-8da2-1dd8b81e514c-kube-api-access-9p95r" (OuterVolumeSpecName: "kube-api-access-9p95r") pod "ec38bdde-60ff-4195-8da2-1dd8b81e514c" (UID: "ec38bdde-60ff-4195-8da2-1dd8b81e514c"). InnerVolumeSpecName "kube-api-access-9p95r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:55:54.066515 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.066456 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ec38bdde-60ff-4195-8da2-1dd8b81e514c" (UID: "ec38bdde-60ff-4195-8da2-1dd8b81e514c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:55:54.164945 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.164908 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:55:54.164945 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.164941 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:55:54.164945 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.164952 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-trusted-ca-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:55:54.164945 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.164961 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:55:54.165213 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.164971 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec38bdde-60ff-4195-8da2-1dd8b81e514c-oauth-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:55:54.165213 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.164979 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec38bdde-60ff-4195-8da2-1dd8b81e514c-console-oauth-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:55:54.165213 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.164987 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p95r\" (UniqueName: \"kubernetes.io/projected/ec38bdde-60ff-4195-8da2-1dd8b81e514c-kube-api-access-9p95r\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:55:54.731470 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.731433 2582 generic.go:356] "Generic (PLEG): container finished" podID="ec38bdde-60ff-4195-8da2-1dd8b81e514c" containerID="ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a" exitCode=0 Jul 31 14:55:54.731947 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.731483 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c686965b-78r6c" Jul 31 14:55:54.731947 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.731493 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c686965b-78r6c" event={"ID":"ec38bdde-60ff-4195-8da2-1dd8b81e514c","Type":"ContainerDied","Data":"ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a"} Jul 31 14:55:54.731947 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.731541 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c686965b-78r6c" event={"ID":"ec38bdde-60ff-4195-8da2-1dd8b81e514c","Type":"ContainerDied","Data":"858c38d2ca9df01852e6eae568272a6e4f1a492ef31a6b28300dbda9ee477927"} Jul 31 14:55:54.731947 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.731587 2582 scope.go:117] "RemoveContainer" containerID="ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a" Jul 31 14:55:54.740888 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.740867 2582 scope.go:117] "RemoveContainer" containerID="ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a" Jul 31 14:55:54.741194 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:55:54.741174 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a\": container with ID starting with ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a not found: ID does not exist" containerID="ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a" Jul 31 14:55:54.741262 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.741201 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a"} err="failed to get container status \"ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a\": rpc error: code = NotFound desc = could not find container \"ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a\": container with ID starting with ef4fcad80c68e034465406a6bc1287945536248d9e6d77cc50f8f04d7726079a not found: ID does not exist" Jul 31 14:55:54.754781 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.754752 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c686965b-78r6c"] Jul 31 14:55:54.759028 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.759004 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c686965b-78r6c"] Jul 31 14:55:54.779077 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:54.779043 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec38bdde-60ff-4195-8da2-1dd8b81e514c" path="/var/lib/kubelet/pods/ec38bdde-60ff-4195-8da2-1dd8b81e514c/volumes" Jul 31 14:55:56.554745 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:56.554697 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:56.645899 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:56.645847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:55:57.067964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.067931 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x"] Jul 31 14:55:57.068285 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.068273 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec38bdde-60ff-4195-8da2-1dd8b81e514c" containerName="console" Jul 31 14:55:57.068331 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.068286 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec38bdde-60ff-4195-8da2-1dd8b81e514c" containerName="console" Jul 31 14:55:57.068363 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.068345 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec38bdde-60ff-4195-8da2-1dd8b81e514c" containerName="console" Jul 31 14:55:57.072928 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.072908 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.076073 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.076055 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 14:55:57.076223 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.076205 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 14:55:57.076911 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.076893 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b4497\"" Jul 31 14:55:57.089983 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.089960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.090066 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.089994 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh87w\" (UniqueName: \"kubernetes.io/projected/10e01ae9-e056-4acd-850f-bf11481a0dc8-kube-api-access-fh87w\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.090066 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.090025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.120204 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.120162 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x"] Jul 31 14:55:57.190666 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.190629 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.190860 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.190684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh87w\" (UniqueName: \"kubernetes.io/projected/10e01ae9-e056-4acd-850f-bf11481a0dc8-kube-api-access-fh87w\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.190860 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.190726 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.191087 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.191063 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.191087 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.191079 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.206199 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.206167 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh87w\" (UniqueName: \"kubernetes.io/projected/10e01ae9-e056-4acd-850f-bf11481a0dc8-kube-api-access-fh87w\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.279336 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.279288 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp"] Jul 31 14:55:57.282932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.282912 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.291412 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.291392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.291510 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.291427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtn84\" (UniqueName: \"kubernetes.io/projected/12620a60-2491-4de4-a383-d39d00ebb322-kube-api-access-xtn84\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.291510 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.291504 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.294907 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.294882 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp"] Jul 31 14:55:57.339394 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.339341 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k"] Jul 31 14:55:57.342661 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.342647 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.353236 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.353213 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k"] Jul 31 14:55:57.382230 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.382203 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:55:57.392675 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.392652 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.392781 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.392692 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtn84\" (UniqueName: \"kubernetes.io/projected/12620a60-2491-4de4-a383-d39d00ebb322-kube-api-access-xtn84\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.392781 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.392744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.392781 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.392771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mg2n\" (UniqueName: \"kubernetes.io/projected/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-kube-api-access-7mg2n\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.392951 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.392823 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.392951 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.392854 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.393163 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.393067 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.393163 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.393075 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.405243 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.405210 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtn84\" (UniqueName: \"kubernetes.io/projected/12620a60-2491-4de4-a383-d39d00ebb322-kube-api-access-xtn84\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.447004 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.446974 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx"] Jul 31 14:55:57.452344 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.452323 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.485417 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.469842 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx"] Jul 31 14:55:57.494268 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.494231 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mg2n\" (UniqueName: \"kubernetes.io/projected/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-kube-api-access-7mg2n\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.494397 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.494294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.494397 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.494326 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tppp\" (UniqueName: \"kubernetes.io/projected/2294e310-6e30-4580-aba1-1d643f4c00bf-kube-api-access-5tppp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.494397 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.494356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.494397 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.494381 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.494631 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.494444 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.494899 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.494875 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.494998 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.494951 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.504336 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.504300 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mg2n\" (UniqueName: \"kubernetes.io/projected/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-kube-api-access-7mg2n\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.551908 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.551885 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x"] Jul 31 14:55:57.554025 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:55:57.554001 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e01ae9_e056_4acd_850f_bf11481a0dc8.slice/crio-2880da434be79295cd39a0622a87149bd93d06037715317c22c068b81a2c8d13 WatchSource:0}: Error finding container 2880da434be79295cd39a0622a87149bd93d06037715317c22c068b81a2c8d13: Status 404 returned error can't find the container with id 2880da434be79295cd39a0622a87149bd93d06037715317c22c068b81a2c8d13 Jul 31 14:55:57.592305 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.592282 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:55:57.595234 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.595213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tppp\" (UniqueName: \"kubernetes.io/projected/2294e310-6e30-4580-aba1-1d643f4c00bf-kube-api-access-5tppp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.595346 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.595243 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.595346 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.595282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.595615 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.595599 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.595666 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.595629 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.621351 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.621322 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tppp\" (UniqueName: \"kubernetes.io/projected/2294e310-6e30-4580-aba1-1d643f4c00bf-kube-api-access-5tppp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.651151 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.651056 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:55:57.743046 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.743017 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp"] Jul 31 14:55:57.745264 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:55:57.745223 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12620a60_2491_4de4_a383_d39d00ebb322.slice/crio-d8a1270bbbd07aa5811df786920c4fd0af00c9ca6361db1a3b1d80c629344fed WatchSource:0}: Error finding container d8a1270bbbd07aa5811df786920c4fd0af00c9ca6361db1a3b1d80c629344fed: Status 404 returned error can't find the container with id d8a1270bbbd07aa5811df786920c4fd0af00c9ca6361db1a3b1d80c629344fed Jul 31 14:55:57.750836 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.750812 2582 generic.go:356] "Generic (PLEG): container finished" podID="10e01ae9-e056-4acd-850f-bf11481a0dc8" containerID="6cbd5ade675e870aebb8f94e5b5e34bb9466adea7a297071c4d6148ad29c6560" exitCode=0 Jul 31 14:55:57.750935 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.750863 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" event={"ID":"10e01ae9-e056-4acd-850f-bf11481a0dc8","Type":"ContainerDied","Data":"6cbd5ade675e870aebb8f94e5b5e34bb9466adea7a297071c4d6148ad29c6560"} Jul 31 14:55:57.750935 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.750903 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" event={"ID":"10e01ae9-e056-4acd-850f-bf11481a0dc8","Type":"ContainerStarted","Data":"2880da434be79295cd39a0622a87149bd93d06037715317c22c068b81a2c8d13"} Jul 31 14:55:57.762642 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.762542 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:55:57.798053 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.797996 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k"] Jul 31 14:55:57.800780 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:55:57.800758 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a25ec5_a63a_4c33_9c1e_db46f9a44b7a.slice/crio-c94190c5d3b97881a19d62fb57701b8b92b2bb40739f8cef94c67b15cdd2b8e3 WatchSource:0}: Error finding container c94190c5d3b97881a19d62fb57701b8b92b2bb40739f8cef94c67b15cdd2b8e3: Status 404 returned error can't find the container with id c94190c5d3b97881a19d62fb57701b8b92b2bb40739f8cef94c67b15cdd2b8e3 Jul 31 14:55:57.928083 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:57.928054 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx"] Jul 31 14:55:57.959359 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:55:57.959334 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2294e310_6e30_4580_aba1_1d643f4c00bf.slice/crio-f041e862099f07356db7be49c9d53a0bc8b54f2ae0ca706b36432fac056bf774 WatchSource:0}: Error finding container f041e862099f07356db7be49c9d53a0bc8b54f2ae0ca706b36432fac056bf774: Status 404 returned error can't find the container with id f041e862099f07356db7be49c9d53a0bc8b54f2ae0ca706b36432fac056bf774 Jul 31 14:55:58.758914 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:58.758869 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" event={"ID":"10e01ae9-e056-4acd-850f-bf11481a0dc8","Type":"ContainerStarted","Data":"3cb4c5b31d938d1174eb03fdd7daac74f9e151b4bd0d3b269708552b82be3dbb"} Jul 31 14:55:58.760370 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:58.760348 2582 generic.go:356] "Generic (PLEG): container finished" podID="2294e310-6e30-4580-aba1-1d643f4c00bf" containerID="5ac01f4c93df0e2e410f8c4e3f8f69a62aef51200ecf83ab991ad6cb61a5bf30" exitCode=0 Jul 31 14:55:58.760480 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:58.760412 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" event={"ID":"2294e310-6e30-4580-aba1-1d643f4c00bf","Type":"ContainerDied","Data":"5ac01f4c93df0e2e410f8c4e3f8f69a62aef51200ecf83ab991ad6cb61a5bf30"} Jul 31 14:55:58.760480 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:58.760443 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" event={"ID":"2294e310-6e30-4580-aba1-1d643f4c00bf","Type":"ContainerStarted","Data":"f041e862099f07356db7be49c9d53a0bc8b54f2ae0ca706b36432fac056bf774"} Jul 31 14:55:58.762048 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:58.762022 2582 generic.go:356] "Generic (PLEG): container finished" podID="a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" containerID="d3c4596eac75d28254fcb88ca5def68ec8b14c8226acfbe7059184addcd433b9" exitCode=0 Jul 31 14:55:58.762163 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:58.762134 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" event={"ID":"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a","Type":"ContainerDied","Data":"d3c4596eac75d28254fcb88ca5def68ec8b14c8226acfbe7059184addcd433b9"} Jul 31 14:55:58.762233 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:58.762172 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" event={"ID":"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a","Type":"ContainerStarted","Data":"c94190c5d3b97881a19d62fb57701b8b92b2bb40739f8cef94c67b15cdd2b8e3"} Jul 31 14:55:58.764073 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:58.764048 2582 generic.go:356] "Generic (PLEG): container finished" podID="12620a60-2491-4de4-a383-d39d00ebb322" containerID="703dab20f8c2bbdd04199b667955505d911f049ab4969a7a7e0375f833bb7c86" exitCode=0 Jul 31 14:55:58.764172 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:58.764092 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" event={"ID":"12620a60-2491-4de4-a383-d39d00ebb322","Type":"ContainerDied","Data":"703dab20f8c2bbdd04199b667955505d911f049ab4969a7a7e0375f833bb7c86"} Jul 31 14:55:58.764172 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:58.764122 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" event={"ID":"12620a60-2491-4de4-a383-d39d00ebb322","Type":"ContainerStarted","Data":"d8a1270bbbd07aa5811df786920c4fd0af00c9ca6361db1a3b1d80c629344fed"} Jul 31 14:55:59.773592 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:59.773535 2582 generic.go:356] "Generic (PLEG): container finished" podID="10e01ae9-e056-4acd-850f-bf11481a0dc8" containerID="3cb4c5b31d938d1174eb03fdd7daac74f9e151b4bd0d3b269708552b82be3dbb" exitCode=0 Jul 31 14:55:59.774046 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:59.773597 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" event={"ID":"10e01ae9-e056-4acd-850f-bf11481a0dc8","Type":"ContainerDied","Data":"3cb4c5b31d938d1174eb03fdd7daac74f9e151b4bd0d3b269708552b82be3dbb"} Jul 31 14:55:59.775984 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:59.775955 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" event={"ID":"2294e310-6e30-4580-aba1-1d643f4c00bf","Type":"ContainerStarted","Data":"b16247665be4f54e25f51582290f2ad5a4facfccd9e9fa40196e584f99dde4f3"} Jul 31 14:55:59.778323 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:55:59.778218 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" event={"ID":"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a","Type":"ContainerStarted","Data":"7c9f38cc8b7b612b88074552044c54535442379d1899d6e6979d164d8803fcb4"} Jul 31 14:56:00.783582 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:00.783531 2582 generic.go:356] "Generic (PLEG): container finished" podID="2294e310-6e30-4580-aba1-1d643f4c00bf" containerID="b16247665be4f54e25f51582290f2ad5a4facfccd9e9fa40196e584f99dde4f3" exitCode=0 Jul 31 14:56:00.783998 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:00.783582 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" event={"ID":"2294e310-6e30-4580-aba1-1d643f4c00bf","Type":"ContainerDied","Data":"b16247665be4f54e25f51582290f2ad5a4facfccd9e9fa40196e584f99dde4f3"} Jul 31 14:56:00.785376 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:00.785351 2582 generic.go:356] "Generic (PLEG): container finished" podID="a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" containerID="7c9f38cc8b7b612b88074552044c54535442379d1899d6e6979d164d8803fcb4" exitCode=0 Jul 31 14:56:00.785489 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:00.785426 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" event={"ID":"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a","Type":"ContainerDied","Data":"7c9f38cc8b7b612b88074552044c54535442379d1899d6e6979d164d8803fcb4"} Jul 31 14:56:00.793446 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:00.793411 2582 generic.go:356] "Generic (PLEG): container finished" podID="12620a60-2491-4de4-a383-d39d00ebb322" containerID="db911fdc9adf24770330eb22ecced78d930aae3ca047d041c5d44b648ec6d6ef" exitCode=0 Jul 31 14:56:00.793587 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:00.793471 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" event={"ID":"12620a60-2491-4de4-a383-d39d00ebb322","Type":"ContainerDied","Data":"db911fdc9adf24770330eb22ecced78d930aae3ca047d041c5d44b648ec6d6ef"} Jul 31 14:56:00.795788 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:00.795767 2582 generic.go:356] "Generic (PLEG): container finished" podID="10e01ae9-e056-4acd-850f-bf11481a0dc8" containerID="66e5e25998a1aadbf5a5eac90c6fa20e111e25950b3d0d9eed3e1cf852dd4819" exitCode=0 Jul 31 14:56:00.795867 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:00.795829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" event={"ID":"10e01ae9-e056-4acd-850f-bf11481a0dc8","Type":"ContainerDied","Data":"66e5e25998a1aadbf5a5eac90c6fa20e111e25950b3d0d9eed3e1cf852dd4819"} Jul 31 14:56:01.803048 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:01.803014 2582 generic.go:356] "Generic (PLEG): container finished" podID="2294e310-6e30-4580-aba1-1d643f4c00bf" containerID="dd9e64d8cbff261c9069456cac40d56cc928a91f88964a832400c29245b0c450" exitCode=0 Jul 31 14:56:01.803485 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:01.803098 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" event={"ID":"2294e310-6e30-4580-aba1-1d643f4c00bf","Type":"ContainerDied","Data":"dd9e64d8cbff261c9069456cac40d56cc928a91f88964a832400c29245b0c450"} Jul 31 14:56:01.805047 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:01.805016 2582 generic.go:356] "Generic (PLEG): container finished" podID="a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" containerID="bde08d97a90d4d0d38397995093a9f92fb62aa6acfad2efd46bd3bd614e5c363" exitCode=0 Jul 31 14:56:01.805164 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:01.805093 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" event={"ID":"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a","Type":"ContainerDied","Data":"bde08d97a90d4d0d38397995093a9f92fb62aa6acfad2efd46bd3bd614e5c363"} Jul 31 14:56:01.807021 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:01.807001 2582 generic.go:356] "Generic (PLEG): container finished" podID="12620a60-2491-4de4-a383-d39d00ebb322" containerID="16fc37eeaafd500c40087d374011b3ae7f2c2ca1522717b547292a1a29df8ca3" exitCode=0 Jul 31 14:56:01.807139 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:01.807070 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" event={"ID":"12620a60-2491-4de4-a383-d39d00ebb322","Type":"ContainerDied","Data":"16fc37eeaafd500c40087d374011b3ae7f2c2ca1522717b547292a1a29df8ca3"} Jul 31 14:56:01.916252 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:01.916222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:01.940747 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:01.940725 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:56:01.988392 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:01.988322 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:02.031261 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.031230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-bundle\") pod \"10e01ae9-e056-4acd-850f-bf11481a0dc8\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " Jul 31 14:56:02.031385 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.031274 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh87w\" (UniqueName: \"kubernetes.io/projected/10e01ae9-e056-4acd-850f-bf11481a0dc8-kube-api-access-fh87w\") pod \"10e01ae9-e056-4acd-850f-bf11481a0dc8\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " Jul 31 14:56:02.031385 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.031301 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-util\") pod \"10e01ae9-e056-4acd-850f-bf11481a0dc8\" (UID: \"10e01ae9-e056-4acd-850f-bf11481a0dc8\") " Jul 31 14:56:02.032056 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.032028 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-bundle" (OuterVolumeSpecName: "bundle") pod "10e01ae9-e056-4acd-850f-bf11481a0dc8" (UID: "10e01ae9-e056-4acd-850f-bf11481a0dc8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:56:02.033271 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.033243 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e01ae9-e056-4acd-850f-bf11481a0dc8-kube-api-access-fh87w" (OuterVolumeSpecName: "kube-api-access-fh87w") pod "10e01ae9-e056-4acd-850f-bf11481a0dc8" (UID: "10e01ae9-e056-4acd-850f-bf11481a0dc8"). InnerVolumeSpecName "kube-api-access-fh87w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:56:02.036598 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.036578 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-util" (OuterVolumeSpecName: "util") pod "10e01ae9-e056-4acd-850f-bf11481a0dc8" (UID: "10e01ae9-e056-4acd-850f-bf11481a0dc8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:56:02.132028 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.131992 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:02.132028 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.132022 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh87w\" (UniqueName: \"kubernetes.io/projected/10e01ae9-e056-4acd-850f-bf11481a0dc8-kube-api-access-fh87w\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:02.132028 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.132033 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10e01ae9-e056-4acd-850f-bf11481a0dc8-util\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:02.813814 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.813785 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" Jul 31 14:56:02.813814 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.813798 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053frn6x" event={"ID":"10e01ae9-e056-4acd-850f-bf11481a0dc8","Type":"ContainerDied","Data":"2880da434be79295cd39a0622a87149bd93d06037715317c22c068b81a2c8d13"} Jul 31 14:56:02.814281 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.813853 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2880da434be79295cd39a0622a87149bd93d06037715317c22c068b81a2c8d13" Jul 31 14:56:02.958449 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:02.958426 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:56:03.006480 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.006456 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:56:03.010316 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.010286 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:56:03.041409 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.041346 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-util\") pod \"12620a60-2491-4de4-a383-d39d00ebb322\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " Jul 31 14:56:03.041409 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.041373 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-util\") pod \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " Jul 31 14:56:03.041409 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.041407 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtn84\" (UniqueName: \"kubernetes.io/projected/12620a60-2491-4de4-a383-d39d00ebb322-kube-api-access-xtn84\") pod \"12620a60-2491-4de4-a383-d39d00ebb322\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " Jul 31 14:56:03.041671 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.041452 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mg2n\" (UniqueName: \"kubernetes.io/projected/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-kube-api-access-7mg2n\") pod \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " Jul 31 14:56:03.041671 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.041481 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-util\") pod \"2294e310-6e30-4580-aba1-1d643f4c00bf\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " Jul 31 14:56:03.041671 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.041533 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-bundle\") pod \"2294e310-6e30-4580-aba1-1d643f4c00bf\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " Jul 31 14:56:03.041671 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.041593 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-bundle\") pod \"12620a60-2491-4de4-a383-d39d00ebb322\" (UID: \"12620a60-2491-4de4-a383-d39d00ebb322\") " Jul 31 14:56:03.041671 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.041624 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tppp\" (UniqueName: \"kubernetes.io/projected/2294e310-6e30-4580-aba1-1d643f4c00bf-kube-api-access-5tppp\") pod \"2294e310-6e30-4580-aba1-1d643f4c00bf\" (UID: \"2294e310-6e30-4580-aba1-1d643f4c00bf\") " Jul 31 14:56:03.041671 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.041657 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-bundle\") pod \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\" (UID: \"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a\") " Jul 31 14:56:03.042332 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.042236 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-bundle" (OuterVolumeSpecName: "bundle") pod "2294e310-6e30-4580-aba1-1d643f4c00bf" (UID: "2294e310-6e30-4580-aba1-1d643f4c00bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:56:03.042435 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.042296 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-bundle" (OuterVolumeSpecName: "bundle") pod "12620a60-2491-4de4-a383-d39d00ebb322" (UID: "12620a60-2491-4de4-a383-d39d00ebb322"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:56:03.042435 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.042353 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-bundle" (OuterVolumeSpecName: "bundle") pod "a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" (UID: "a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:56:03.043831 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.043804 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12620a60-2491-4de4-a383-d39d00ebb322-kube-api-access-xtn84" (OuterVolumeSpecName: "kube-api-access-xtn84") pod "12620a60-2491-4de4-a383-d39d00ebb322" (UID: "12620a60-2491-4de4-a383-d39d00ebb322"). InnerVolumeSpecName "kube-api-access-xtn84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:56:03.044063 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.044034 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-kube-api-access-7mg2n" (OuterVolumeSpecName: "kube-api-access-7mg2n") pod "a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" (UID: "a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a"). InnerVolumeSpecName "kube-api-access-7mg2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:56:03.044222 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.044199 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2294e310-6e30-4580-aba1-1d643f4c00bf-kube-api-access-5tppp" (OuterVolumeSpecName: "kube-api-access-5tppp") pod "2294e310-6e30-4580-aba1-1d643f4c00bf" (UID: "2294e310-6e30-4580-aba1-1d643f4c00bf"). InnerVolumeSpecName "kube-api-access-5tppp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:56:03.047298 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.047277 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-util" (OuterVolumeSpecName: "util") pod "2294e310-6e30-4580-aba1-1d643f4c00bf" (UID: "2294e310-6e30-4580-aba1-1d643f4c00bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:56:03.142689 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.142668 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-util\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:03.142689 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.142687 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2294e310-6e30-4580-aba1-1d643f4c00bf-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:03.142835 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.142698 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:03.142835 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.142706 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tppp\" (UniqueName: \"kubernetes.io/projected/2294e310-6e30-4580-aba1-1d643f4c00bf-kube-api-access-5tppp\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:03.142835 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.142716 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:03.142835 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.142726 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtn84\" (UniqueName: \"kubernetes.io/projected/12620a60-2491-4de4-a383-d39d00ebb322-kube-api-access-xtn84\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:03.142835 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.142736 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mg2n\" (UniqueName: \"kubernetes.io/projected/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-kube-api-access-7mg2n\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:03.159870 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.159844 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-util" (OuterVolumeSpecName: "util") pod "a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" (UID: "a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:56:03.162445 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.162423 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-util" (OuterVolumeSpecName: "util") pod "12620a60-2491-4de4-a383-d39d00ebb322" (UID: "12620a60-2491-4de4-a383-d39d00ebb322"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:56:03.243463 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.243441 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12620a60-2491-4de4-a383-d39d00ebb322-util\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:03.243463 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.243460 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a-util\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:56:03.820712 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.820673 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" event={"ID":"2294e310-6e30-4580-aba1-1d643f4c00bf","Type":"ContainerDied","Data":"f041e862099f07356db7be49c9d53a0bc8b54f2ae0ca706b36432fac056bf774"} Jul 31 14:56:03.821142 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.820723 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f041e862099f07356db7be49c9d53a0bc8b54f2ae0ca706b36432fac056bf774" Jul 31 14:56:03.821142 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.820681 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbszhx" Jul 31 14:56:03.822722 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.822703 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" Jul 31 14:56:03.822851 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.822725 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a04x25k" event={"ID":"a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a","Type":"ContainerDied","Data":"c94190c5d3b97881a19d62fb57701b8b92b2bb40739f8cef94c67b15cdd2b8e3"} Jul 31 14:56:03.822851 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.822764 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c94190c5d3b97881a19d62fb57701b8b92b2bb40739f8cef94c67b15cdd2b8e3" Jul 31 14:56:03.824600 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.824581 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" event={"ID":"12620a60-2491-4de4-a383-d39d00ebb322","Type":"ContainerDied","Data":"d8a1270bbbd07aa5811df786920c4fd0af00c9ca6361db1a3b1d80c629344fed"} Jul 31 14:56:03.824671 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.824608 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czh8sp" Jul 31 14:56:03.824724 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:03.824613 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8a1270bbbd07aa5811df786920c4fd0af00c9ca6361db1a3b1d80c629344fed" Jul 31 14:56:06.021378 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:06.021347 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:06.869708 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:06.869678 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:09.263610 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263577 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-7grjx"] Jul 31 14:56:09.263964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263900 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" containerName="pull" Jul 31 14:56:09.263964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263911 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" containerName="pull" Jul 31 14:56:09.263964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263924 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e01ae9-e056-4acd-850f-bf11481a0dc8" containerName="util" Jul 31 14:56:09.263964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263929 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e01ae9-e056-4acd-850f-bf11481a0dc8" containerName="util" Jul 31 14:56:09.263964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263935 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2294e310-6e30-4580-aba1-1d643f4c00bf" containerName="util" Jul 31 14:56:09.263964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263941 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2294e310-6e30-4580-aba1-1d643f4c00bf" containerName="util" Jul 31 14:56:09.263964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263949 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e01ae9-e056-4acd-850f-bf11481a0dc8" containerName="extract" Jul 31 14:56:09.263964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263954 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e01ae9-e056-4acd-850f-bf11481a0dc8" containerName="extract" Jul 31 14:56:09.263964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263962 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12620a60-2491-4de4-a383-d39d00ebb322" containerName="extract" Jul 31 14:56:09.263964 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263967 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12620a60-2491-4de4-a383-d39d00ebb322" containerName="extract" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263975 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e01ae9-e056-4acd-850f-bf11481a0dc8" containerName="pull" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263980 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e01ae9-e056-4acd-850f-bf11481a0dc8" containerName="pull" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263986 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2294e310-6e30-4580-aba1-1d643f4c00bf" containerName="extract" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.263992 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2294e310-6e30-4580-aba1-1d643f4c00bf" containerName="extract" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264004 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2294e310-6e30-4580-aba1-1d643f4c00bf" containerName="pull" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264009 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2294e310-6e30-4580-aba1-1d643f4c00bf" containerName="pull" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264015 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12620a60-2491-4de4-a383-d39d00ebb322" containerName="pull" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264020 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12620a60-2491-4de4-a383-d39d00ebb322" containerName="pull" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264025 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" containerName="extract" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264030 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" containerName="extract" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264039 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12620a60-2491-4de4-a383-d39d00ebb322" containerName="util" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264044 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12620a60-2491-4de4-a383-d39d00ebb322" containerName="util" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264050 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" containerName="util" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264055 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" containerName="util" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264105 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6a25ec5-a63a-4c33-9c1e-db46f9a44b7a" containerName="extract" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264112 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e01ae9-e056-4acd-850f-bf11481a0dc8" containerName="extract" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264118 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2294e310-6e30-4580-aba1-1d643f4c00bf" containerName="extract" Jul 31 14:56:09.264256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.264124 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="12620a60-2491-4de4-a383-d39d00ebb322" containerName="extract" Jul 31 14:56:09.274111 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.274089 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-7grjx" Jul 31 14:56:09.277669 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.277645 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mlml7\"" Jul 31 14:56:09.278440 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.278419 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 14:56:09.278440 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.278432 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 14:56:09.281125 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.281099 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-7grjx"] Jul 31 14:56:09.291725 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.291701 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7g62\" (UniqueName: \"kubernetes.io/projected/1980debd-6ca2-42d6-943a-246d406d5b32-kube-api-access-k7g62\") pod \"authorino-operator-56fb6f6856-7grjx\" (UID: \"1980debd-6ca2-42d6-943a-246d406d5b32\") " pod="kuadrant-system/authorino-operator-56fb6f6856-7grjx" Jul 31 14:56:09.392859 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.392828 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7g62\" (UniqueName: \"kubernetes.io/projected/1980debd-6ca2-42d6-943a-246d406d5b32-kube-api-access-k7g62\") pod \"authorino-operator-56fb6f6856-7grjx\" (UID: \"1980debd-6ca2-42d6-943a-246d406d5b32\") " pod="kuadrant-system/authorino-operator-56fb6f6856-7grjx" Jul 31 14:56:09.405020 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.404993 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7g62\" (UniqueName: \"kubernetes.io/projected/1980debd-6ca2-42d6-943a-246d406d5b32-kube-api-access-k7g62\") pod \"authorino-operator-56fb6f6856-7grjx\" (UID: \"1980debd-6ca2-42d6-943a-246d406d5b32\") " pod="kuadrant-system/authorino-operator-56fb6f6856-7grjx" Jul 31 14:56:09.586198 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.586117 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-7grjx" Jul 31 14:56:09.674358 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.674332 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:09.761342 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:56:09.761314 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1980debd_6ca2_42d6_943a_246d406d5b32.slice/crio-a9d098eac1351cffa2d26417d9ac86fe8469bd8705ea2658f038c1a27ca0a80e WatchSource:0}: Error finding container a9d098eac1351cffa2d26417d9ac86fe8469bd8705ea2658f038c1a27ca0a80e: Status 404 returned error can't find the container with id a9d098eac1351cffa2d26417d9ac86fe8469bd8705ea2658f038c1a27ca0a80e Jul 31 14:56:09.763224 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.763123 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-7grjx"] Jul 31 14:56:09.777432 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.777400 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:09.853523 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.853458 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-7grjx" event={"ID":"1980debd-6ca2-42d6-943a-246d406d5b32","Type":"ContainerStarted","Data":"a9d098eac1351cffa2d26417d9ac86fe8469bd8705ea2658f038c1a27ca0a80e"} Jul 31 14:56:09.896529 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:09.896498 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:11.079574 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:11.079538 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:11.431682 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:11.431646 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b"] Jul 31 14:56:11.440482 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:11.440458 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b" Jul 31 14:56:11.443694 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:11.443673 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 31 14:56:11.443834 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:11.443817 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qvvg5\"" Jul 31 14:56:11.460169 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:11.460143 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b"] Jul 31 14:56:11.512706 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:11.512671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zxl\" (UniqueName: \"kubernetes.io/projected/044753fc-40f4-405c-8e9d-7a7d3bffa5d1-kube-api-access-24zxl\") pod \"dns-operator-controller-manager-ccbdf686-lkd9b\" (UID: \"044753fc-40f4-405c-8e9d-7a7d3bffa5d1\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b" Jul 31 14:56:11.613740 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:11.613701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zxl\" (UniqueName: \"kubernetes.io/projected/044753fc-40f4-405c-8e9d-7a7d3bffa5d1-kube-api-access-24zxl\") pod \"dns-operator-controller-manager-ccbdf686-lkd9b\" (UID: \"044753fc-40f4-405c-8e9d-7a7d3bffa5d1\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b" Jul 31 14:56:11.635414 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:11.635378 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zxl\" (UniqueName: \"kubernetes.io/projected/044753fc-40f4-405c-8e9d-7a7d3bffa5d1-kube-api-access-24zxl\") pod \"dns-operator-controller-manager-ccbdf686-lkd9b\" (UID: \"044753fc-40f4-405c-8e9d-7a7d3bffa5d1\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b" Jul 31 14:56:11.752487 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:11.752403 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b" Jul 31 14:56:12.136230 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:12.136200 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b"] Jul 31 14:56:12.140500 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:56:12.140475 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044753fc_40f4_405c_8e9d_7a7d3bffa5d1.slice/crio-dac37c20e8d6fafec423e362ff93b4b221a35a60d214dc5b98efcb55b5202e62 WatchSource:0}: Error finding container dac37c20e8d6fafec423e362ff93b4b221a35a60d214dc5b98efcb55b5202e62: Status 404 returned error can't find the container with id dac37c20e8d6fafec423e362ff93b4b221a35a60d214dc5b98efcb55b5202e62 Jul 31 14:56:12.222036 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:12.222004 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:12.461611 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:12.461579 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:12.871938 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:12.871843 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b" event={"ID":"044753fc-40f4-405c-8e9d-7a7d3bffa5d1","Type":"ContainerStarted","Data":"dac37c20e8d6fafec423e362ff93b4b221a35a60d214dc5b98efcb55b5202e62"} Jul 31 14:56:12.873326 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:12.873301 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-7grjx" event={"ID":"1980debd-6ca2-42d6-943a-246d406d5b32","Type":"ContainerStarted","Data":"c3ad7202d4dd98c758a62436455e4e5f94d848da99d1a932ad70e4a135aa8c41"} Jul 31 14:56:12.873432 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:12.873411 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-7grjx" Jul 31 14:56:12.902676 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:12.902628 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-7grjx" podStartSLOduration=1.618831301 podStartE2EDuration="3.902613657s" totalImagesPullingTime="2.283782356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:12.902176161 +0000 UTC m=+484.503612499" watchObservedRunningTime="2026-07-31 14:56:12.902613657 +0000 UTC m=+484.504049993" Jul 31 14:56:13.822502 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:13.822471 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4"] Jul 31 14:56:13.827112 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:13.827093 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4" Jul 31 14:56:13.829813 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:13.829798 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-glzmf\"" Jul 31 14:56:13.844334 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:13.844307 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4"] Jul 31 14:56:13.880147 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:13.880112 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b" event={"ID":"044753fc-40f4-405c-8e9d-7a7d3bffa5d1","Type":"ContainerStarted","Data":"06c3bf02b36dde6a03402223c4b37ea9083175a8ab65c7f1646b693bbec9169f"} Jul 31 14:56:13.880265 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:13.880197 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b" Jul 31 14:56:13.909471 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:13.909418 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b" podStartSLOduration=1.2620328490000001 podStartE2EDuration="2.909400881s" totalImagesPullingTime="1.647368032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:13.908732587 +0000 UTC m=+485.510168927" watchObservedRunningTime="2026-07-31 14:56:13.909400881 +0000 UTC m=+485.510837218" Jul 31 14:56:13.931295 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:13.931262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbbm5\" (UniqueName: \"kubernetes.io/projected/32cf4f4f-23b5-4ad9-9e48-4da4cb9dac94-kube-api-access-fbbm5\") pod \"limitador-operator-controller-manager-774d94476f-zwvm4\" (UID: \"32cf4f4f-23b5-4ad9-9e48-4da4cb9dac94\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4" Jul 31 14:56:14.032708 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:14.032669 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbbm5\" (UniqueName: \"kubernetes.io/projected/32cf4f4f-23b5-4ad9-9e48-4da4cb9dac94-kube-api-access-fbbm5\") pod \"limitador-operator-controller-manager-774d94476f-zwvm4\" (UID: \"32cf4f4f-23b5-4ad9-9e48-4da4cb9dac94\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4" Jul 31 14:56:14.042991 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:14.042923 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbbm5\" (UniqueName: \"kubernetes.io/projected/32cf4f4f-23b5-4ad9-9e48-4da4cb9dac94-kube-api-access-fbbm5\") pod \"limitador-operator-controller-manager-774d94476f-zwvm4\" (UID: \"32cf4f4f-23b5-4ad9-9e48-4da4cb9dac94\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4" Jul 31 14:56:14.154478 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:14.154441 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4" Jul 31 14:56:14.216948 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:14.216885 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:14.299616 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:14.299587 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4"] Jul 31 14:56:14.622433 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:14.622349 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:14.886005 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:14.885963 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4" event={"ID":"32cf4f4f-23b5-4ad9-9e48-4da4cb9dac94","Type":"ContainerStarted","Data":"ab48003ba1712f030a4fcebf33dca89d51fd21655c7ecac9135e356a4739c2be"} Jul 31 14:56:15.053401 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:15.053367 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:15.315624 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:15.315530 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:15.415477 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:15.415445 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:16.899613 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:16.899581 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4" event={"ID":"32cf4f4f-23b5-4ad9-9e48-4da4cb9dac94","Type":"ContainerStarted","Data":"331c7d9a5d58f73f98f57848b24431f80e8e6aa76121a2ecad7c2a8bf5742270"} Jul 31 14:56:16.899972 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:16.899692 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4" Jul 31 14:56:16.919774 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:16.919704 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4" podStartSLOduration=2.203040897 podStartE2EDuration="3.919687704s" totalImagesPullingTime="1.716646807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:16.918369396 +0000 UTC m=+488.519805734" watchObservedRunningTime="2026-07-31 14:56:16.919687704 +0000 UTC m=+488.521124042" Jul 31 14:56:20.186132 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:20.186098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:20.283435 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:20.283402 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:20.449740 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:20.449668 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:20.553808 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:20.553777 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:20.855069 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:20.854980 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:20.966500 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:20.966473 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:23.882517 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:23.882489 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-7grjx" Jul 31 14:56:24.034189 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:24.034161 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:24.140001 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:24.139933 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:24.888549 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:24.888520 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-lkd9b" Jul 31 14:56:24.984735 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:24.984701 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:26.047296 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:26.047265 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:27.907210 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:27.907179 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-zwvm4" Jul 31 14:56:28.270212 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:28.270144 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:28.375280 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:28.375253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:33.012671 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:33.012637 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:33.116202 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:33.116169 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:33.188205 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:33.188177 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:38.436350 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:38.436320 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:40.701465 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:40.701427 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:40.852334 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:40.852300 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:40.953551 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:40.953467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:41.447869 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:41.447840 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:41.548511 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:41.548482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:43.848968 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:43.848938 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:43.948344 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:43.948307 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:49.059242 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:49.059194 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:49.165474 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:49.165446 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:52.855765 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:52.855736 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:52.962081 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:52.962049 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:53.054252 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:56:53.054223 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:06.632711 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.632632 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:06.643767 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.643739 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-h9bvt"] Jul 31 14:57:06.647120 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.647103 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:06.649801 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.649782 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 31 14:57:06.649921 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.649832 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ffvtl\"" Jul 31 14:57:06.651591 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.651572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d03d396c-8933-4693-9933-55db68416af6-config-file\") pod \"limitador-limitador-859c4d96d6-h9bvt\" (UID: \"d03d396c-8933-4693-9933-55db68416af6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:06.651694 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.651608 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ptd\" (UniqueName: \"kubernetes.io/projected/d03d396c-8933-4693-9933-55db68416af6-kube-api-access-v2ptd\") pod \"limitador-limitador-859c4d96d6-h9bvt\" (UID: \"d03d396c-8933-4693-9933-55db68416af6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:06.657763 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.657742 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-h9bvt"] Jul 31 14:57:06.732601 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.728883 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:06.732601 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.729205 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-h9bvt"] Jul 31 14:57:06.752020 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.751983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d03d396c-8933-4693-9933-55db68416af6-config-file\") pod \"limitador-limitador-859c4d96d6-h9bvt\" (UID: \"d03d396c-8933-4693-9933-55db68416af6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:06.752020 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.752024 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ptd\" (UniqueName: \"kubernetes.io/projected/d03d396c-8933-4693-9933-55db68416af6-kube-api-access-v2ptd\") pod \"limitador-limitador-859c4d96d6-h9bvt\" (UID: \"d03d396c-8933-4693-9933-55db68416af6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:06.752684 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.752664 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d03d396c-8933-4693-9933-55db68416af6-config-file\") pod \"limitador-limitador-859c4d96d6-h9bvt\" (UID: \"d03d396c-8933-4693-9933-55db68416af6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:06.761393 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.761365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ptd\" (UniqueName: \"kubernetes.io/projected/d03d396c-8933-4693-9933-55db68416af6-kube-api-access-v2ptd\") pod \"limitador-limitador-859c4d96d6-h9bvt\" (UID: \"d03d396c-8933-4693-9933-55db68416af6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:06.825006 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.824968 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:06.958988 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:06.958902 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:07.102444 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:07.102421 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-h9bvt"] Jul 31 14:57:07.104739 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:57:07.104713 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03d396c_8933_4693_9933_55db68416af6.slice/crio-019b1fbe4c2e0937e77fc9c1e0565a9af0daf79d064c877426106a04c5b17b54 WatchSource:0}: Error finding container 019b1fbe4c2e0937e77fc9c1e0565a9af0daf79d064c877426106a04c5b17b54: Status 404 returned error can't find the container with id 019b1fbe4c2e0937e77fc9c1e0565a9af0daf79d064c877426106a04c5b17b54 Jul 31 14:57:07.155421 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:07.155391 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" event={"ID":"d03d396c-8933-4693-9933-55db68416af6","Type":"ContainerStarted","Data":"019b1fbe4c2e0937e77fc9c1e0565a9af0daf79d064c877426106a04c5b17b54"} Jul 31 14:57:07.487546 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:07.487508 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:07.616508 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:07.616474 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:09.167684 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:09.167646 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" event={"ID":"d03d396c-8933-4693-9933-55db68416af6","Type":"ContainerStarted","Data":"da3a68ba0fbbadcb232c7aec199915e04a77d4e3ef647930ab4af76639a7e60a"} Jul 31 14:57:09.168150 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:09.167752 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:09.192743 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:09.192695 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" podStartSLOduration=1.799177925 podStartE2EDuration="3.192680627s" totalImagesPullingTime="1.393502702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:09.190000032 +0000 UTC m=+540.791436361" watchObservedRunningTime="2026-07-31 14:57:09.192680627 +0000 UTC m=+540.794116983" Jul 31 14:57:20.174216 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:20.174184 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:20.739722 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:20.739692 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-h9bvt"] Jul 31 14:57:20.739885 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:20.739867 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" podUID="d03d396c-8933-4693-9933-55db68416af6" containerName="limitador" containerID="cri-o://da3a68ba0fbbadcb232c7aec199915e04a77d4e3ef647930ab4af76639a7e60a" gracePeriod=30 Jul 31 14:57:21.230301 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:21.230266 2582 generic.go:356] "Generic (PLEG): container finished" podID="d03d396c-8933-4693-9933-55db68416af6" containerID="da3a68ba0fbbadcb232c7aec199915e04a77d4e3ef647930ab4af76639a7e60a" exitCode=0 Jul 31 14:57:21.230651 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:21.230343 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" event={"ID":"d03d396c-8933-4693-9933-55db68416af6","Type":"ContainerDied","Data":"da3a68ba0fbbadcb232c7aec199915e04a77d4e3ef647930ab4af76639a7e60a"} Jul 31 14:57:21.284018 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:21.283998 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:21.366525 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:21.366469 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2ptd\" (UniqueName: \"kubernetes.io/projected/d03d396c-8933-4693-9933-55db68416af6-kube-api-access-v2ptd\") pod \"d03d396c-8933-4693-9933-55db68416af6\" (UID: \"d03d396c-8933-4693-9933-55db68416af6\") " Jul 31 14:57:21.366525 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:21.366516 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d03d396c-8933-4693-9933-55db68416af6-config-file\") pod \"d03d396c-8933-4693-9933-55db68416af6\" (UID: \"d03d396c-8933-4693-9933-55db68416af6\") " Jul 31 14:57:21.366885 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:21.366866 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03d396c-8933-4693-9933-55db68416af6-config-file" (OuterVolumeSpecName: "config-file") pod "d03d396c-8933-4693-9933-55db68416af6" (UID: "d03d396c-8933-4693-9933-55db68416af6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:57:21.368486 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:21.368464 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03d396c-8933-4693-9933-55db68416af6-kube-api-access-v2ptd" (OuterVolumeSpecName: "kube-api-access-v2ptd") pod "d03d396c-8933-4693-9933-55db68416af6" (UID: "d03d396c-8933-4693-9933-55db68416af6"). InnerVolumeSpecName "kube-api-access-v2ptd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:57:21.468034 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:21.468010 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ptd\" (UniqueName: \"kubernetes.io/projected/d03d396c-8933-4693-9933-55db68416af6-kube-api-access-v2ptd\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:57:21.468034 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:21.468032 2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d03d396c-8933-4693-9933-55db68416af6-config-file\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:57:22.237364 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:22.237331 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" Jul 31 14:57:22.237887 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:22.237360 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-h9bvt" event={"ID":"d03d396c-8933-4693-9933-55db68416af6","Type":"ContainerDied","Data":"019b1fbe4c2e0937e77fc9c1e0565a9af0daf79d064c877426106a04c5b17b54"} Jul 31 14:57:22.237887 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:22.237448 2582 scope.go:117] "RemoveContainer" containerID="da3a68ba0fbbadcb232c7aec199915e04a77d4e3ef647930ab4af76639a7e60a" Jul 31 14:57:22.260212 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:22.260188 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-h9bvt"] Jul 31 14:57:22.263816 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:22.263791 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-h9bvt"] Jul 31 14:57:22.776629 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:22.776590 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d03d396c-8933-4693-9933-55db68416af6" path="/var/lib/kubelet/pods/d03d396c-8933-4693-9933-55db68416af6/volumes" Jul 31 14:57:26.842838 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:26.842801 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:27.033145 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:27.033101 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:35.803788 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:35.803753 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:35.903895 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:35.903864 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:52.489000 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.488964 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps"] Jul 31 14:57:52.489502 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.489474 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d03d396c-8933-4693-9933-55db68416af6" containerName="limitador" Jul 31 14:57:52.489502 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.489493 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d03d396c-8933-4693-9933-55db68416af6" containerName="limitador" Jul 31 14:57:52.489665 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.489641 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d03d396c-8933-4693-9933-55db68416af6" containerName="limitador" Jul 31 14:57:52.493686 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.493665 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.510819 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.510795 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps"] Jul 31 14:57:52.639566 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.639531 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfml\" (UniqueName: \"kubernetes.io/projected/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-kube-api-access-slfml\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.639732 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.639620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.639732 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.639641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.639732 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.639664 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.639732 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.639686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.639871 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.639765 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.639871 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.639794 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.740680 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.740601 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.740680 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.740638 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.740902 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.740682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfml\" (UniqueName: \"kubernetes.io/projected/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-kube-api-access-slfml\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.740902 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.740732 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.740902 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.740749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.740902 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.740769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.740902 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.740848 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.741338 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.741312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.743216 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.743187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.743313 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.743219 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.743466 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.743449 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.743508 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.743450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.752154 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.752129 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfml\" (UniqueName: \"kubernetes.io/projected/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-kube-api-access-slfml\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.752333 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.752313 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9b2557-cc06-4cd1-bce2-3c650fd86bc3-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-csdps\" (UID: \"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.802222 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.802195 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:52.943716 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.943685 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps"] Jul 31 14:57:52.944464 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:57:52.944443 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9b2557_cc06_4cd1_bce2_3c650fd86bc3.slice/crio-0107de78f8e3fe4cea138f2e211c1866fcdbbcb8dfce435295e8d8e8fc46cd41 WatchSource:0}: Error finding container 0107de78f8e3fe4cea138f2e211c1866fcdbbcb8dfce435295e8d8e8fc46cd41: Status 404 returned error can't find the container with id 0107de78f8e3fe4cea138f2e211c1866fcdbbcb8dfce435295e8d8e8fc46cd41 Jul 31 14:57:52.946364 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.946331 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 14:57:52.946457 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:52.946387 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 14:57:53.398750 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:53.398719 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" event={"ID":"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3","Type":"ContainerStarted","Data":"cee48c34e149b112e16a47eb62571b031a2d3cd6644adbccf6202168b1cb149a"} Jul 31 14:57:53.398942 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:53.398770 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" event={"ID":"2d9b2557-cc06-4cd1-bce2-3c650fd86bc3","Type":"ContainerStarted","Data":"0107de78f8e3fe4cea138f2e211c1866fcdbbcb8dfce435295e8d8e8fc46cd41"} Jul 31 14:57:53.399060 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:53.399036 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:53.400528 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:53.400507 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-csdps container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 14:57:53.400643 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:53.400549 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" podUID="2d9b2557-cc06-4cd1-bce2-3c650fd86bc3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 14:57:53.422242 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:53.422187 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" podStartSLOduration=1.422165774 podStartE2EDuration="1.422165774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:53.421514411 +0000 UTC m=+585.022950751" watchObservedRunningTime="2026-07-31 14:57:53.422165774 +0000 UTC m=+585.023602112" Jul 31 14:57:54.404332 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.404301 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-csdps" Jul 31 14:57:54.464920 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.464885 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs"] Jul 31 14:57:54.465194 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.465158 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" podUID="5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" containerName="discovery" containerID="cri-o://740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623" gracePeriod=30 Jul 31 14:57:54.712077 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.712055 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:57:54.859733 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.859695 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-cacerts\") pod \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " Jul 31 14:57:54.859911 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.859744 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-ca-configmap\") pod \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " Jul 31 14:57:54.859911 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.859837 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-kubeconfig\") pod \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " Jul 31 14:57:54.859911 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.859859 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-local-certs\") pod \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " Jul 31 14:57:54.859911 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.859885 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-token\") pod \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " Jul 31 14:57:54.860108 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.859918 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-dns-cert\") pod \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " Jul 31 14:57:54.860108 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.859949 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr4sw\" (UniqueName: \"kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-kube-api-access-xr4sw\") pod \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\" (UID: \"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68\") " Jul 31 14:57:54.860245 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.860215 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" (UID: "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:57:54.860372 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.860350 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-ca-configmap\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:57:54.862475 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.862451 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" (UID: "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:57:54.862635 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.862515 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" (UID: "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:57:54.862635 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.862613 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-local-certs" (OuterVolumeSpecName: "local-certs") pod "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" (UID: "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:57:54.862765 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.862676 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-kube-api-access-xr4sw" (OuterVolumeSpecName: "kube-api-access-xr4sw") pod "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" (UID: "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68"). InnerVolumeSpecName "kube-api-access-xr4sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:57:54.862765 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.862751 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-token" (OuterVolumeSpecName: "istio-token") pod "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" (UID: "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:57:54.862838 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.862822 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-cacerts" (OuterVolumeSpecName: "cacerts") pod "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" (UID: "5dd029c9-b3e4-41d2-bbec-9e11f4a7af68"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:57:54.961753 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.961669 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-csr-dns-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:57:54.961753 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.961695 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr4sw\" (UniqueName: \"kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-kube-api-access-xr4sw\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:57:54.961753 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.961705 2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-cacerts\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:57:54.961753 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.961715 2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-kubeconfig\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:57:54.961753 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.961724 2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-local-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:57:54.961753 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:54.961732 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68-istio-token\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:57:55.410049 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:55.410018 2582 generic.go:356] "Generic (PLEG): container finished" podID="5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" containerID="740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623" exitCode=0 Jul 31 14:57:55.410514 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:55.410065 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" Jul 31 14:57:55.410514 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:55.410108 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" event={"ID":"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68","Type":"ContainerDied","Data":"740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623"} Jul 31 14:57:55.410514 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:55.410166 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" event={"ID":"5dd029c9-b3e4-41d2-bbec-9e11f4a7af68","Type":"ContainerDied","Data":"4864e30a2fcb3001fc7d92add3eff3c11d69efd8693b480eb1a2fd749ddd978a"} Jul 31 14:57:55.410514 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:55.410197 2582 scope.go:117] "RemoveContainer" containerID="740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623" Jul 31 14:57:55.419234 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:55.419219 2582 scope.go:117] "RemoveContainer" containerID="740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623" Jul 31 14:57:55.419472 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:57:55.419452 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623\": container with ID starting with 740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623 not found: ID does not exist" containerID="740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623" Jul 31 14:57:55.419578 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:55.419481 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623"} err="failed to get container status \"740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623\": rpc error: code = NotFound desc = could not find container \"740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623\": container with ID starting with 740cf7df3485474241e0eaeb4bcbe38334a2b07a25678e2dbbb91100eb977623 not found: ID does not exist" Jul 31 14:57:55.438584 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:55.438544 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs"] Jul 31 14:57:55.462497 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:55.462468 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs"] Jul 31 14:57:56.775862 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:56.775827 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" path="/var/lib/kubelet/pods/5dd029c9-b3e4-41d2-bbec-9e11f4a7af68/volumes" Jul 31 14:57:59.678331 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:59.678289 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-mw7hs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 31 14:57:59.678891 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:57:59.678354 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw7hs" podUID="5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.32:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 31 14:58:07.908467 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:07.908438 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:08.006951 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.006913 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-vntjc"] Jul 31 14:58:08.007336 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.007321 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" containerName="discovery" Jul 31 14:58:08.007383 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.007338 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" containerName="discovery" Jul 31 14:58:08.007418 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.007398 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd029c9-b3e4-41d2-bbec-9e11f4a7af68" containerName="discovery" Jul 31 14:58:08.010294 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.010273 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:08.012926 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.012904 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-8h775\"" Jul 31 14:58:08.012926 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.012920 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 31 14:58:08.013120 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.013067 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 31 14:58:08.013585 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.013550 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 31 14:58:08.026526 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.026505 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-vntjc"] Jul 31 14:58:08.035584 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.035544 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-68988fb7b6-5dmnc"] Jul 31 14:58:08.039392 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.039375 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:08.042641 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.042621 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 31 14:58:08.042791 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.042726 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-ds25s\"" Jul 31 14:58:08.055166 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.055141 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-swhnw"] Jul 31 14:58:08.058355 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.058335 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68988fb7b6-5dmnc"] Jul 31 14:58:08.058460 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.058434 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-swhnw" Jul 31 14:58:08.060946 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.060924 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jg8zm\"" Jul 31 14:58:08.061423 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.061407 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 31 14:58:08.080185 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.080147 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8nk\" (UniqueName: \"kubernetes.io/projected/219152d6-7cdb-4f36-9c93-0fc8330411a4-kube-api-access-jq8nk\") pod \"llmisvc-controller-manager-68988fb7b6-5dmnc\" (UID: \"219152d6-7cdb-4f36-9c93-0fc8330411a4\") " pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:08.080349 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.080227 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkpcq\" (UniqueName: \"kubernetes.io/projected/61904792-0b6c-462b-988a-d92f9862699b-kube-api-access-gkpcq\") pod \"seaweedfs-64568bcd49-swhnw\" (UID: \"61904792-0b6c-462b-988a-d92f9862699b\") " pod="kserve/seaweedfs-64568bcd49-swhnw" Jul 31 14:58:08.080349 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.080258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/219152d6-7cdb-4f36-9c93-0fc8330411a4-cert\") pod \"llmisvc-controller-manager-68988fb7b6-5dmnc\" (UID: \"219152d6-7cdb-4f36-9c93-0fc8330411a4\") " pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:08.080349 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.080319 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/61904792-0b6c-462b-988a-d92f9862699b-data\") pod \"seaweedfs-64568bcd49-swhnw\" (UID: \"61904792-0b6c-462b-988a-d92f9862699b\") " pod="kserve/seaweedfs-64568bcd49-swhnw" Jul 31 14:58:08.080526 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.080406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e28939af-5c1b-4b4a-821c-40aa39c38f80-cert\") pod \"kserve-controller-manager-fcc9dd4db-vntjc\" (UID: \"e28939af-5c1b-4b4a-821c-40aa39c38f80\") " pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:08.080599 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.080529 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdtdx\" (UniqueName: \"kubernetes.io/projected/e28939af-5c1b-4b4a-821c-40aa39c38f80-kube-api-access-fdtdx\") pod \"kserve-controller-manager-fcc9dd4db-vntjc\" (UID: \"e28939af-5c1b-4b4a-821c-40aa39c38f80\") " pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:08.081376 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.081351 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-swhnw"] Jul 31 14:58:08.181628 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.181526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/61904792-0b6c-462b-988a-d92f9862699b-data\") pod \"seaweedfs-64568bcd49-swhnw\" (UID: \"61904792-0b6c-462b-988a-d92f9862699b\") " pod="kserve/seaweedfs-64568bcd49-swhnw" Jul 31 14:58:08.181628 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.181583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e28939af-5c1b-4b4a-821c-40aa39c38f80-cert\") pod \"kserve-controller-manager-fcc9dd4db-vntjc\" (UID: \"e28939af-5c1b-4b4a-821c-40aa39c38f80\") " pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:08.181628 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.181615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdtdx\" (UniqueName: \"kubernetes.io/projected/e28939af-5c1b-4b4a-821c-40aa39c38f80-kube-api-access-fdtdx\") pod \"kserve-controller-manager-fcc9dd4db-vntjc\" (UID: \"e28939af-5c1b-4b4a-821c-40aa39c38f80\") " pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:08.181848 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.181656 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8nk\" (UniqueName: \"kubernetes.io/projected/219152d6-7cdb-4f36-9c93-0fc8330411a4-kube-api-access-jq8nk\") pod \"llmisvc-controller-manager-68988fb7b6-5dmnc\" (UID: \"219152d6-7cdb-4f36-9c93-0fc8330411a4\") " pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:08.181848 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.181691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkpcq\" (UniqueName: \"kubernetes.io/projected/61904792-0b6c-462b-988a-d92f9862699b-kube-api-access-gkpcq\") pod \"seaweedfs-64568bcd49-swhnw\" (UID: \"61904792-0b6c-462b-988a-d92f9862699b\") " pod="kserve/seaweedfs-64568bcd49-swhnw" Jul 31 14:58:08.181848 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.181706 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/219152d6-7cdb-4f36-9c93-0fc8330411a4-cert\") pod \"llmisvc-controller-manager-68988fb7b6-5dmnc\" (UID: \"219152d6-7cdb-4f36-9c93-0fc8330411a4\") " pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:08.181848 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:58:08.181807 2582 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 31 14:58:08.181998 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:58:08.181879 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/219152d6-7cdb-4f36-9c93-0fc8330411a4-cert podName:219152d6-7cdb-4f36-9c93-0fc8330411a4 nodeName:}" failed. No retries permitted until 2026-07-31 14:58:08.681860094 +0000 UTC m=+600.283296409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/219152d6-7cdb-4f36-9c93-0fc8330411a4-cert") pod "llmisvc-controller-manager-68988fb7b6-5dmnc" (UID: "219152d6-7cdb-4f36-9c93-0fc8330411a4") : secret "llmisvc-webhook-server-cert" not found Jul 31 14:58:08.181998 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.181968 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/61904792-0b6c-462b-988a-d92f9862699b-data\") pod \"seaweedfs-64568bcd49-swhnw\" (UID: \"61904792-0b6c-462b-988a-d92f9862699b\") " pod="kserve/seaweedfs-64568bcd49-swhnw" Jul 31 14:58:08.184055 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.184036 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e28939af-5c1b-4b4a-821c-40aa39c38f80-cert\") pod \"kserve-controller-manager-fcc9dd4db-vntjc\" (UID: \"e28939af-5c1b-4b4a-821c-40aa39c38f80\") " pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:08.207394 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.207360 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdtdx\" (UniqueName: \"kubernetes.io/projected/e28939af-5c1b-4b4a-821c-40aa39c38f80-kube-api-access-fdtdx\") pod \"kserve-controller-manager-fcc9dd4db-vntjc\" (UID: \"e28939af-5c1b-4b4a-821c-40aa39c38f80\") " pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:08.208285 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.208266 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkpcq\" (UniqueName: \"kubernetes.io/projected/61904792-0b6c-462b-988a-d92f9862699b-kube-api-access-gkpcq\") pod \"seaweedfs-64568bcd49-swhnw\" (UID: \"61904792-0b6c-462b-988a-d92f9862699b\") " pod="kserve/seaweedfs-64568bcd49-swhnw" Jul 31 14:58:08.208539 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.208521 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8nk\" (UniqueName: \"kubernetes.io/projected/219152d6-7cdb-4f36-9c93-0fc8330411a4-kube-api-access-jq8nk\") pod \"llmisvc-controller-manager-68988fb7b6-5dmnc\" (UID: \"219152d6-7cdb-4f36-9c93-0fc8330411a4\") " pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:08.232359 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.232332 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:08.321521 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.321495 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:08.344041 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.344007 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:08.368080 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.368051 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-swhnw" Jul 31 14:58:08.468918 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.468892 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-vntjc"] Jul 31 14:58:08.470396 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:58:08.470367 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28939af_5c1b_4b4a_821c_40aa39c38f80.slice/crio-4b5fc46af2b8d7b16d2f60bd494b864a1ef57deb302d98a137277242d3a2c7ad WatchSource:0}: Error finding container 4b5fc46af2b8d7b16d2f60bd494b864a1ef57deb302d98a137277242d3a2c7ad: Status 404 returned error can't find the container with id 4b5fc46af2b8d7b16d2f60bd494b864a1ef57deb302d98a137277242d3a2c7ad Jul 31 14:58:08.483747 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.483712 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" event={"ID":"e28939af-5c1b-4b4a-821c-40aa39c38f80","Type":"ContainerStarted","Data":"4b5fc46af2b8d7b16d2f60bd494b864a1ef57deb302d98a137277242d3a2c7ad"} Jul 31 14:58:08.516987 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.516967 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-swhnw"] Jul 31 14:58:08.518404 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:58:08.518380 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61904792_0b6c_462b_988a_d92f9862699b.slice/crio-a49779eaaf9d0af670db8764df5dab76e7fb6e62ccf341d91ef338bd027ef391 WatchSource:0}: Error finding container a49779eaaf9d0af670db8764df5dab76e7fb6e62ccf341d91ef338bd027ef391: Status 404 returned error can't find the container with id a49779eaaf9d0af670db8764df5dab76e7fb6e62ccf341d91ef338bd027ef391 Jul 31 14:58:08.685440 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.685412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/219152d6-7cdb-4f36-9c93-0fc8330411a4-cert\") pod \"llmisvc-controller-manager-68988fb7b6-5dmnc\" (UID: \"219152d6-7cdb-4f36-9c93-0fc8330411a4\") " pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:08.687650 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.687632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/219152d6-7cdb-4f36-9c93-0fc8330411a4-cert\") pod \"llmisvc-controller-manager-68988fb7b6-5dmnc\" (UID: \"219152d6-7cdb-4f36-9c93-0fc8330411a4\") " pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:08.697684 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.697658 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:08.797420 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.797397 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:08.896222 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.896188 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:08.952218 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.952198 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-ds25s\"" Jul 31 14:58:08.960280 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:08.960261 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:09.093591 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:09.093548 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68988fb7b6-5dmnc"] Jul 31 14:58:09.095287 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:58:09.095256 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod219152d6_7cdb_4f36_9c93_0fc8330411a4.slice/crio-a4d09b05db00c1f2ad9f7cac1a9d8884982f8d8e8fbd3a467cb9b1fb74a0d797 WatchSource:0}: Error finding container a4d09b05db00c1f2ad9f7cac1a9d8884982f8d8e8fbd3a467cb9b1fb74a0d797: Status 404 returned error can't find the container with id a4d09b05db00c1f2ad9f7cac1a9d8884982f8d8e8fbd3a467cb9b1fb74a0d797 Jul 31 14:58:09.493094 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:09.493012 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-swhnw" event={"ID":"61904792-0b6c-462b-988a-d92f9862699b","Type":"ContainerStarted","Data":"a49779eaaf9d0af670db8764df5dab76e7fb6e62ccf341d91ef338bd027ef391"} Jul 31 14:58:09.494552 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:09.494520 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" event={"ID":"219152d6-7cdb-4f36-9c93-0fc8330411a4","Type":"ContainerStarted","Data":"a4d09b05db00c1f2ad9f7cac1a9d8884982f8d8e8fbd3a467cb9b1fb74a0d797"} Jul 31 14:58:13.506234 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:13.506204 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 31 14:58:14.526048 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:14.526012 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-swhnw" event={"ID":"61904792-0b6c-462b-988a-d92f9862699b","Type":"ContainerStarted","Data":"e157b480a69bfeda2b75119cc4c4557873eab82b24954d10146af6edc05b6f8c"} Jul 31 14:58:14.529721 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:14.526065 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-swhnw" Jul 31 14:58:14.529721 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:14.527500 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" event={"ID":"e28939af-5c1b-4b4a-821c-40aa39c38f80","Type":"ContainerStarted","Data":"e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b"} Jul 31 14:58:14.529721 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:14.527547 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:14.529721 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:14.528804 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" event={"ID":"219152d6-7cdb-4f36-9c93-0fc8330411a4","Type":"ContainerStarted","Data":"a320adcf98a12117070da5fe5d61a24794f04c313ac071acbebe81cca734e10a"} Jul 31 14:58:14.529721 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:14.528917 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:14.548685 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:14.548635 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-swhnw" podStartSLOduration=1.564759921 podStartE2EDuration="6.548619686s" totalImagesPullingTime="4.983859765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:58:14.546315784 +0000 UTC m=+606.147752123" watchObservedRunningTime="2026-07-31 14:58:14.548619686 +0000 UTC m=+606.150056022" Jul 31 14:58:14.565171 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:14.565125 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" podStartSLOduration=3.512274507 podStartE2EDuration="7.56511339s" totalImagesPullingTime="4.052838883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:58:14.563411402 +0000 UTC m=+606.164847744" watchObservedRunningTime="2026-07-31 14:58:14.56511339 +0000 UTC m=+606.166549727" Jul 31 14:58:14.583003 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:14.582959 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" podStartSLOduration=2.125057599 podStartE2EDuration="6.582943918s" totalImagesPullingTime="4.457886319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:58:14.582009101 +0000 UTC m=+606.183445438" watchObservedRunningTime="2026-07-31 14:58:14.582943918 +0000 UTC m=+606.184380256" Jul 31 14:58:20.537269 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:20.537237 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-swhnw" Jul 31 14:58:45.537221 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:45.537186 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-68988fb7b6-5dmnc" Jul 31 14:58:45.540117 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:45.540095 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:49.590576 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:49.590521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:54.307488 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.307450 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-vntjc"] Jul 31 14:58:54.307973 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.307762 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" podUID="e28939af-5c1b-4b4a-821c-40aa39c38f80" containerName="manager" containerID="cri-o://e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b" gracePeriod=10 Jul 31 14:58:54.334804 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.334779 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-qg2xt"] Jul 31 14:58:54.338133 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.338118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" Jul 31 14:58:54.349339 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.349316 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-qg2xt"] Jul 31 14:58:54.376892 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.376866 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ba4ed2f-b509-43d2-8585-ea8724354232-cert\") pod \"kserve-controller-manager-fcc9dd4db-qg2xt\" (UID: \"4ba4ed2f-b509-43d2-8585-ea8724354232\") " pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" Jul 31 14:58:54.377011 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.376907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rww5p\" (UniqueName: \"kubernetes.io/projected/4ba4ed2f-b509-43d2-8585-ea8724354232-kube-api-access-rww5p\") pod \"kserve-controller-manager-fcc9dd4db-qg2xt\" (UID: \"4ba4ed2f-b509-43d2-8585-ea8724354232\") " pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" Jul 31 14:58:54.478542 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.478511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ba4ed2f-b509-43d2-8585-ea8724354232-cert\") pod \"kserve-controller-manager-fcc9dd4db-qg2xt\" (UID: \"4ba4ed2f-b509-43d2-8585-ea8724354232\") " pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" Jul 31 14:58:54.478694 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.478608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rww5p\" (UniqueName: \"kubernetes.io/projected/4ba4ed2f-b509-43d2-8585-ea8724354232-kube-api-access-rww5p\") pod \"kserve-controller-manager-fcc9dd4db-qg2xt\" (UID: \"4ba4ed2f-b509-43d2-8585-ea8724354232\") " pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" Jul 31 14:58:54.480852 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.480827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ba4ed2f-b509-43d2-8585-ea8724354232-cert\") pod \"kserve-controller-manager-fcc9dd4db-qg2xt\" (UID: \"4ba4ed2f-b509-43d2-8585-ea8724354232\") " pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" Jul 31 14:58:54.487337 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.487311 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rww5p\" (UniqueName: \"kubernetes.io/projected/4ba4ed2f-b509-43d2-8585-ea8724354232-kube-api-access-rww5p\") pod \"kserve-controller-manager-fcc9dd4db-qg2xt\" (UID: \"4ba4ed2f-b509-43d2-8585-ea8724354232\") " pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" Jul 31 14:58:54.552201 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.552180 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:54.579001 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.578926 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdtdx\" (UniqueName: \"kubernetes.io/projected/e28939af-5c1b-4b4a-821c-40aa39c38f80-kube-api-access-fdtdx\") pod \"e28939af-5c1b-4b4a-821c-40aa39c38f80\" (UID: \"e28939af-5c1b-4b4a-821c-40aa39c38f80\") " Jul 31 14:58:54.579147 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.579018 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e28939af-5c1b-4b4a-821c-40aa39c38f80-cert\") pod \"e28939af-5c1b-4b4a-821c-40aa39c38f80\" (UID: \"e28939af-5c1b-4b4a-821c-40aa39c38f80\") " Jul 31 14:58:54.581264 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.581229 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28939af-5c1b-4b4a-821c-40aa39c38f80-kube-api-access-fdtdx" (OuterVolumeSpecName: "kube-api-access-fdtdx") pod "e28939af-5c1b-4b4a-821c-40aa39c38f80" (UID: "e28939af-5c1b-4b4a-821c-40aa39c38f80"). InnerVolumeSpecName "kube-api-access-fdtdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:58:54.581404 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.581360 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e28939af-5c1b-4b4a-821c-40aa39c38f80-cert" (OuterVolumeSpecName: "cert") pod "e28939af-5c1b-4b4a-821c-40aa39c38f80" (UID: "e28939af-5c1b-4b4a-821c-40aa39c38f80"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:58:54.680335 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.680305 2582 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e28939af-5c1b-4b4a-821c-40aa39c38f80-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:58:54.680335 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.680328 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdtdx\" (UniqueName: \"kubernetes.io/projected/e28939af-5c1b-4b4a-821c-40aa39c38f80-kube-api-access-fdtdx\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 14:58:54.694234 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.694208 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" Jul 31 14:58:54.740862 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.740817 2582 generic.go:356] "Generic (PLEG): container finished" podID="e28939af-5c1b-4b4a-821c-40aa39c38f80" containerID="e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b" exitCode=0 Jul 31 14:58:54.741018 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.740870 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" Jul 31 14:58:54.741018 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.740914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" event={"ID":"e28939af-5c1b-4b4a-821c-40aa39c38f80","Type":"ContainerDied","Data":"e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b"} Jul 31 14:58:54.741018 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.740992 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-vntjc" event={"ID":"e28939af-5c1b-4b4a-821c-40aa39c38f80","Type":"ContainerDied","Data":"4b5fc46af2b8d7b16d2f60bd494b864a1ef57deb302d98a137277242d3a2c7ad"} Jul 31 14:58:54.741172 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.741027 2582 scope.go:117] "RemoveContainer" containerID="e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b" Jul 31 14:58:54.753717 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.753695 2582 scope.go:117] "RemoveContainer" containerID="e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b" Jul 31 14:58:54.754086 ip-10-0-135-143 kubenswrapper[2582]: E0731 14:58:54.754060 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b\": container with ID starting with e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b not found: ID does not exist" containerID="e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b" Jul 31 14:58:54.754188 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.754097 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b"} err="failed to get container status \"e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b\": rpc error: code = NotFound desc = could not find container \"e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b\": container with ID starting with e3adc0717e5a9cd4f7519596fa4c2f48fdd88ee17de01c85d26029bc555bc24b not found: ID does not exist" Jul 31 14:58:54.766402 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.766379 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-vntjc"] Jul 31 14:58:54.775158 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.775131 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-vntjc"] Jul 31 14:58:54.828490 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.828464 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-qg2xt"] Jul 31 14:58:54.830044 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:58:54.829986 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba4ed2f_b509_43d2_8585_ea8724354232.slice/crio-db64393c8c47e3bcfe9a522e5fba1a9f46d3def6ec68a290b6e8337f20170a29 WatchSource:0}: Error finding container db64393c8c47e3bcfe9a522e5fba1a9f46d3def6ec68a290b6e8337f20170a29: Status 404 returned error can't find the container with id db64393c8c47e3bcfe9a522e5fba1a9f46d3def6ec68a290b6e8337f20170a29 Jul 31 14:58:54.831256 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:54.831242 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:58:55.747966 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:55.747938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" event={"ID":"4ba4ed2f-b509-43d2-8585-ea8724354232","Type":"ContainerStarted","Data":"d0cafb3417b42309f0159721f956111493b011aae00afcc2119e5390c1c1bdc2"} Jul 31 14:58:55.748347 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:55.747985 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" event={"ID":"4ba4ed2f-b509-43d2-8585-ea8724354232","Type":"ContainerStarted","Data":"db64393c8c47e3bcfe9a522e5fba1a9f46d3def6ec68a290b6e8337f20170a29"} Jul 31 14:58:55.748347 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:55.748012 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" Jul 31 14:58:55.766853 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:55.766811 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" podStartSLOduration=1.446895545 podStartE2EDuration="1.766766425s" totalImagesPullingTime="319.87088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:58:55.765691574 +0000 UTC m=+647.367127911" watchObservedRunningTime="2026-07-31 14:58:55.766766425 +0000 UTC m=+647.368202761" Jul 31 14:58:56.774574 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:58:56.774526 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e28939af-5c1b-4b4a-821c-40aa39c38f80" path="/var/lib/kubelet/pods/e28939af-5c1b-4b4a-821c-40aa39c38f80/volumes" Jul 31 14:59:26.758325 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:26.758292 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-fcc9dd4db-qg2xt" Jul 31 14:59:27.753361 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.753327 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:27.813593 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.813566 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-xxkbm"] Jul 31 14:59:27.813971 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.813917 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28939af-5c1b-4b4a-821c-40aa39c38f80" containerName="manager" Jul 31 14:59:27.813971 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.813928 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28939af-5c1b-4b4a-821c-40aa39c38f80" containerName="manager" Jul 31 14:59:27.814047 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.814001 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28939af-5c1b-4b4a-821c-40aa39c38f80" containerName="manager" Jul 31 14:59:27.817064 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.817046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-xxkbm" Jul 31 14:59:27.819890 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.819873 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Jul 31 14:59:27.819890 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.819886 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-b9qgw\"" Jul 31 14:59:27.838060 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.838035 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-xxkbm"] Jul 31 14:59:27.849411 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.849390 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:27.917249 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.917225 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:27.942409 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.942388 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def362b-7157-45ac-9a3c-0e2cf890a4af-tls-certs\") pod \"model-serving-api-7b954ffc95-xxkbm\" (UID: \"4def362b-7157-45ac-9a3c-0e2cf890a4af\") " pod="kserve/model-serving-api-7b954ffc95-xxkbm" Jul 31 14:59:27.942526 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:27.942489 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4l4\" (UniqueName: \"kubernetes.io/projected/4def362b-7157-45ac-9a3c-0e2cf890a4af-kube-api-access-8n4l4\") pod \"model-serving-api-7b954ffc95-xxkbm\" (UID: \"4def362b-7157-45ac-9a3c-0e2cf890a4af\") " pod="kserve/model-serving-api-7b954ffc95-xxkbm" Jul 31 14:59:28.017600 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.017527 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:28.043111 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.043089 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def362b-7157-45ac-9a3c-0e2cf890a4af-tls-certs\") pod \"model-serving-api-7b954ffc95-xxkbm\" (UID: \"4def362b-7157-45ac-9a3c-0e2cf890a4af\") " pod="kserve/model-serving-api-7b954ffc95-xxkbm" Jul 31 14:59:28.043205 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.043114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4l4\" (UniqueName: \"kubernetes.io/projected/4def362b-7157-45ac-9a3c-0e2cf890a4af-kube-api-access-8n4l4\") pod \"model-serving-api-7b954ffc95-xxkbm\" (UID: \"4def362b-7157-45ac-9a3c-0e2cf890a4af\") " pod="kserve/model-serving-api-7b954ffc95-xxkbm" Jul 31 14:59:28.045521 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.045501 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def362b-7157-45ac-9a3c-0e2cf890a4af-tls-certs\") pod \"model-serving-api-7b954ffc95-xxkbm\" (UID: \"4def362b-7157-45ac-9a3c-0e2cf890a4af\") " pod="kserve/model-serving-api-7b954ffc95-xxkbm" Jul 31 14:59:28.054188 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.054167 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4l4\" (UniqueName: \"kubernetes.io/projected/4def362b-7157-45ac-9a3c-0e2cf890a4af-kube-api-access-8n4l4\") pod \"model-serving-api-7b954ffc95-xxkbm\" (UID: \"4def362b-7157-45ac-9a3c-0e2cf890a4af\") " pod="kserve/model-serving-api-7b954ffc95-xxkbm" Jul 31 14:59:28.127074 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.127053 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-xxkbm" Jul 31 14:59:28.229166 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.229134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:28.285626 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.285603 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-xxkbm"] Jul 31 14:59:28.287989 ip-10-0-135-143 kubenswrapper[2582]: W0731 14:59:28.287953 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4def362b_7157_45ac_9a3c_0e2cf890a4af.slice/crio-b2a3c6638f9c474eab3d83d2d6bab7842765033e6c816535e07f8e3fede7bb12 WatchSource:0}: Error finding container b2a3c6638f9c474eab3d83d2d6bab7842765033e6c816535e07f8e3fede7bb12: Status 404 returned error can't find the container with id b2a3c6638f9c474eab3d83d2d6bab7842765033e6c816535e07f8e3fede7bb12 Jul 31 14:59:28.563029 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.562941 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:28.664224 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.664162 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:28.925897 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:28.925862 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-xxkbm" event={"ID":"4def362b-7157-45ac-9a3c-0e2cf890a4af","Type":"ContainerStarted","Data":"b2a3c6638f9c474eab3d83d2d6bab7842765033e6c816535e07f8e3fede7bb12"} Jul 31 14:59:29.932531 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:29.932501 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-xxkbm" event={"ID":"4def362b-7157-45ac-9a3c-0e2cf890a4af","Type":"ContainerStarted","Data":"d57f2b1037a157459adbe89040b41f687088817a33bb38fd64dedc1d1fc9152d"} Jul 31 14:59:29.932932 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:29.932628 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-xxkbm" Jul 31 14:59:29.958416 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:29.956794 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-xxkbm" podStartSLOduration=1.617254835 podStartE2EDuration="2.956776067s" totalImagesPullingTime="1.339521232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:59:29.954971271 +0000 UTC m=+681.556407608" watchObservedRunningTime="2026-07-31 14:59:29.956776067 +0000 UTC m=+681.558212405" Jul 31 14:59:34.975335 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:34.975301 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:40.941348 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:40.941321 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-xxkbm" Jul 31 14:59:56.522332 ip-10-0-135-143 kubenswrapper[2582]: I0731 14:59:56.522294 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:01.655605 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:01.655548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:01.758837 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:01.758794 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:14.247577 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:14.247491 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:14.341141 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:14.341102 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:24.038753 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.038703 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:24.380192 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.380162 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj"] Jul 31 15:01:24.383919 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.383899 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:01:24.386382 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.386362 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\"" Jul 31 15:01:24.387373 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.387349 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\"" Jul 31 15:01:24.387464 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.387382 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\"" Jul 31 15:01:24.387464 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.387361 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-jn852\"" Jul 31 15:01:24.387548 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.387418 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-6gkrp\"" Jul 31 15:01:24.392184 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.392160 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj"] Jul 31 15:01:24.407969 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.407949 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:24.471625 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.471598 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvv7\" (UniqueName: \"kubernetes.io/projected/a649b9de-4d3c-4a94-9d49-008d62b474d0-kube-api-access-zxvv7\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj\" (UID: \"a649b9de-4d3c-4a94-9d49-008d62b474d0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:01:24.471746 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.471643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a649b9de-4d3c-4a94-9d49-008d62b474d0-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj\" (UID: \"a649b9de-4d3c-4a94-9d49-008d62b474d0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:01:24.572227 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.572198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvv7\" (UniqueName: \"kubernetes.io/projected/a649b9de-4d3c-4a94-9d49-008d62b474d0-kube-api-access-zxvv7\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj\" (UID: \"a649b9de-4d3c-4a94-9d49-008d62b474d0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:01:24.572370 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.572243 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a649b9de-4d3c-4a94-9d49-008d62b474d0-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj\" (UID: \"a649b9de-4d3c-4a94-9d49-008d62b474d0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:01:24.574573 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.574544 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a649b9de-4d3c-4a94-9d49-008d62b474d0-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj\" (UID: \"a649b9de-4d3c-4a94-9d49-008d62b474d0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:01:24.579843 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.579819 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvv7\" (UniqueName: \"kubernetes.io/projected/a649b9de-4d3c-4a94-9d49-008d62b474d0-kube-api-access-zxvv7\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj\" (UID: \"a649b9de-4d3c-4a94-9d49-008d62b474d0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:01:24.694405 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.694322 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:01:24.830734 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:24.830704 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj"] Jul 31 15:01:24.831887 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:01:24.831855 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda649b9de_4d3c_4a94_9d49_008d62b474d0.slice/crio-540e7af3959e93182a48089ff1bd0f44e210765287017d2081a136ab69de4983 WatchSource:0}: Error finding container 540e7af3959e93182a48089ff1bd0f44e210765287017d2081a136ab69de4983: Status 404 returned error can't find the container with id 540e7af3959e93182a48089ff1bd0f44e210765287017d2081a136ab69de4983 Jul 31 15:01:25.552344 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:25.552302 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" event={"ID":"a649b9de-4d3c-4a94-9d49-008d62b474d0","Type":"ContainerStarted","Data":"540e7af3959e93182a48089ff1bd0f44e210765287017d2081a136ab69de4983"} Jul 31 15:01:28.234777 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.234745 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:28.408729 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.408696 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:28.550209 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.550139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:28.573321 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.573288 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" event={"ID":"a649b9de-4d3c-4a94-9d49-008d62b474d0","Type":"ContainerStarted","Data":"613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138"} Jul 31 15:01:28.573497 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.573457 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:01:28.602592 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.602514 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" podStartSLOduration=1.8834870129999999 podStartE2EDuration="4.602495809s" totalImagesPullingTime="2.719008796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:01:28.599170152 +0000 UTC m=+800.200606485" watchObservedRunningTime="2026-07-31 15:01:28.602495809 +0000 UTC m=+800.203932147" Jul 31 15:01:28.693216 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.693183 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh"] Jul 31 15:01:28.696937 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.696912 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:01:28.700178 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.700156 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-kx6m8\"" Jul 31 15:01:28.701811 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.701752 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\"" Jul 31 15:01:28.701949 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.701896 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\"" Jul 31 15:01:28.702071 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.702047 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-czjml\"" Jul 31 15:01:28.702199 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.702049 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\"" Jul 31 15:01:28.712092 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.712067 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh"] Jul 31 15:01:28.712394 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.712374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjmr\" (UniqueName: \"kubernetes.io/projected/fc56912c-23c7-47da-82e0-8801c843203a-kube-api-access-zkjmr\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh\" (UID: \"fc56912c-23c7-47da-82e0-8801c843203a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:01:28.712448 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.712416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc56912c-23c7-47da-82e0-8801c843203a-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh\" (UID: \"fc56912c-23c7-47da-82e0-8801c843203a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:01:28.739120 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.739087 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:28.813440 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.813367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc56912c-23c7-47da-82e0-8801c843203a-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh\" (UID: \"fc56912c-23c7-47da-82e0-8801c843203a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:01:28.813604 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.813460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjmr\" (UniqueName: \"kubernetes.io/projected/fc56912c-23c7-47da-82e0-8801c843203a-kube-api-access-zkjmr\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh\" (UID: \"fc56912c-23c7-47da-82e0-8801c843203a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:01:28.815804 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.815779 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc56912c-23c7-47da-82e0-8801c843203a-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh\" (UID: \"fc56912c-23c7-47da-82e0-8801c843203a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:01:28.822368 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:28.822340 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjmr\" (UniqueName: \"kubernetes.io/projected/fc56912c-23c7-47da-82e0-8801c843203a-kube-api-access-zkjmr\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh\" (UID: \"fc56912c-23c7-47da-82e0-8801c843203a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:01:29.009729 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:29.009696 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:01:29.148659 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:29.148633 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh"] Jul 31 15:01:29.150338 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:01:29.150307 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc56912c_23c7_47da_82e0_8801c843203a.slice/crio-321bafd865f7a82d55ebbb28ca8be5b5867323885b58fec5adc7083788621890 WatchSource:0}: Error finding container 321bafd865f7a82d55ebbb28ca8be5b5867323885b58fec5adc7083788621890: Status 404 returned error can't find the container with id 321bafd865f7a82d55ebbb28ca8be5b5867323885b58fec5adc7083788621890 Jul 31 15:01:29.580493 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:29.580452 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" event={"ID":"fc56912c-23c7-47da-82e0-8801c843203a","Type":"ContainerStarted","Data":"c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517"} Jul 31 15:01:29.580922 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:29.580582 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:01:29.580922 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:29.580628 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" event={"ID":"fc56912c-23c7-47da-82e0-8801c843203a","Type":"ContainerStarted","Data":"321bafd865f7a82d55ebbb28ca8be5b5867323885b58fec5adc7083788621890"} Jul 31 15:01:29.605776 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:29.605728 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" podStartSLOduration=1.6057118190000002 podStartE2EDuration="1.605711819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:01:29.604523275 +0000 UTC m=+801.205959622" watchObservedRunningTime="2026-07-31 15:01:29.605711819 +0000 UTC m=+801.207148158" Jul 31 15:01:40.926655 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:40.926616 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:49.728870 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:49.728836 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:59.582274 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:01:59.582243 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:02:00.587663 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:00.587635 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:02:06.432416 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:06.432384 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj"] Jul 31 15:02:06.436947 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:06.436915 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:06.528182 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:06.528147 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:09.582629 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:09.582588 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" podUID="a649b9de-4d3c-4a94-9d49-008d62b474d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:02:12.190514 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:12.190469 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:19.582361 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:19.582312 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" podUID="a649b9de-4d3c-4a94-9d49-008d62b474d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:02:21.454692 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.454639 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" podUID="a649b9de-4d3c-4a94-9d49-008d62b474d0" containerName="main" containerID="cri-o://613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138" gracePeriod=45 Jul 31 15:02:21.702845 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.702823 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:02:21.765551 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.765481 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a649b9de-4d3c-4a94-9d49-008d62b474d0-tls-certs\") pod \"a649b9de-4d3c-4a94-9d49-008d62b474d0\" (UID: \"a649b9de-4d3c-4a94-9d49-008d62b474d0\") " Jul 31 15:02:21.765551 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.765537 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxvv7\" (UniqueName: \"kubernetes.io/projected/a649b9de-4d3c-4a94-9d49-008d62b474d0-kube-api-access-zxvv7\") pod \"a649b9de-4d3c-4a94-9d49-008d62b474d0\" (UID: \"a649b9de-4d3c-4a94-9d49-008d62b474d0\") " Jul 31 15:02:21.767596 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.767572 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a649b9de-4d3c-4a94-9d49-008d62b474d0-kube-api-access-zxvv7" (OuterVolumeSpecName: "kube-api-access-zxvv7") pod "a649b9de-4d3c-4a94-9d49-008d62b474d0" (UID: "a649b9de-4d3c-4a94-9d49-008d62b474d0"). InnerVolumeSpecName "kube-api-access-zxvv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:02:21.767696 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.767635 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a649b9de-4d3c-4a94-9d49-008d62b474d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a649b9de-4d3c-4a94-9d49-008d62b474d0" (UID: "a649b9de-4d3c-4a94-9d49-008d62b474d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:02:21.865881 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.865844 2582 generic.go:356] "Generic (PLEG): container finished" podID="a649b9de-4d3c-4a94-9d49-008d62b474d0" containerID="613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138" exitCode=0 Jul 31 15:02:21.866033 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.865891 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" Jul 31 15:02:21.866033 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.865932 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" event={"ID":"a649b9de-4d3c-4a94-9d49-008d62b474d0","Type":"ContainerDied","Data":"613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138"} Jul 31 15:02:21.866033 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.866003 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj" event={"ID":"a649b9de-4d3c-4a94-9d49-008d62b474d0","Type":"ContainerDied","Data":"540e7af3959e93182a48089ff1bd0f44e210765287017d2081a136ab69de4983"} Jul 31 15:02:21.866139 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.866033 2582 scope.go:117] "RemoveContainer" containerID="613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138" Jul 31 15:02:21.866188 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.866143 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a649b9de-4d3c-4a94-9d49-008d62b474d0-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:02:21.866299 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.866279 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxvv7\" (UniqueName: \"kubernetes.io/projected/a649b9de-4d3c-4a94-9d49-008d62b474d0-kube-api-access-zxvv7\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:02:21.875644 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.875628 2582 scope.go:117] "RemoveContainer" containerID="613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138" Jul 31 15:02:21.875872 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:02:21.875851 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138\": container with ID starting with 613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138 not found: ID does not exist" containerID="613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138" Jul 31 15:02:21.875920 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.875880 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138"} err="failed to get container status \"613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138\": rpc error: code = NotFound desc = could not find container \"613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138\": container with ID starting with 613a33d0be48a45b95ac4ce157a36722b5bc384ad60404395f7a484546e63138 not found: ID does not exist" Jul 31 15:02:21.888035 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.888013 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj"] Jul 31 15:02:21.891622 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:21.891604 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-txqlj"] Jul 31 15:02:22.776085 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:22.776048 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a649b9de-4d3c-4a94-9d49-008d62b474d0" path="/var/lib/kubelet/pods/a649b9de-4d3c-4a94-9d49-008d62b474d0/volumes" Jul 31 15:02:26.697865 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:26.697831 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:26.797891 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:26.797860 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:31.521015 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.520960 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6"] Jul 31 15:02:31.521720 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.521695 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a649b9de-4d3c-4a94-9d49-008d62b474d0" containerName="main" Jul 31 15:02:31.521941 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.521918 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a649b9de-4d3c-4a94-9d49-008d62b474d0" containerName="main" Jul 31 15:02:31.522075 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.522051 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a649b9de-4d3c-4a94-9d49-008d62b474d0" containerName="main" Jul 31 15:02:31.526694 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.526674 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.529580 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.529549 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\"" Jul 31 15:02:31.530791 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.530737 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-fwhrc\"" Jul 31 15:02:31.530888 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.530796 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\"" Jul 31 15:02:31.530888 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.530815 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\"" Jul 31 15:02:31.538022 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.538004 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6"] Jul 31 15:02:31.654772 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.654728 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd774962-974f-4b94-867d-2f1381fc74c0-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.654978 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.654783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.654978 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.654907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.654978 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.654939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.654978 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.654968 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-home\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.655180 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.655051 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfjqw\" (UniqueName: \"kubernetes.io/projected/fd774962-974f-4b94-867d-2f1381fc74c0-kube-api-access-vfjqw\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.755977 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.755936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd774962-974f-4b94-867d-2f1381fc74c0-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.756167 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.755995 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.756167 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.756078 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.756167 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.756104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.756167 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.756132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-home\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.756387 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.756200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfjqw\" (UniqueName: \"kubernetes.io/projected/fd774962-974f-4b94-867d-2f1381fc74c0-kube-api-access-vfjqw\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.756498 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.756472 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.756608 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.756532 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-home\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.756733 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.756710 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.758489 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.758453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.758713 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.758691 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd774962-974f-4b94-867d-2f1381fc74c0-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.770234 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.770210 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfjqw\" (UniqueName: \"kubernetes.io/projected/fd774962-974f-4b94-867d-2f1381fc74c0-kube-api-access-vfjqw\") pod \"fc-concurrency-test-kserve-b5d44467c-xfxq6\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.838228 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.838136 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:31.980822 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:31.980791 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6"] Jul 31 15:02:31.982335 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:02:31.982309 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd774962_974f_4b94_867d_2f1381fc74c0.slice/crio-ee76e9496ea575ffeeb6bafb0a97fca9f3be42331683a75d27444200b7052172 WatchSource:0}: Error finding container ee76e9496ea575ffeeb6bafb0a97fca9f3be42331683a75d27444200b7052172: Status 404 returned error can't find the container with id ee76e9496ea575ffeeb6bafb0a97fca9f3be42331683a75d27444200b7052172 Jul 31 15:02:32.932619 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:32.932581 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" event={"ID":"fd774962-974f-4b94-867d-2f1381fc74c0","Type":"ContainerStarted","Data":"ee76e9496ea575ffeeb6bafb0a97fca9f3be42331683a75d27444200b7052172"} Jul 31 15:02:33.202090 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:33.201992 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:33.303228 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:33.303190 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:33.681022 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:33.680979 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:33.941737 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:33.941701 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" event={"ID":"fd774962-974f-4b94-867d-2f1381fc74c0","Type":"ContainerStarted","Data":"c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50"} Jul 31 15:02:33.963312 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:33.963253 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" podStartSLOduration=1.526926312 podStartE2EDuration="2.963237087s" totalImagesPullingTime="1.436310775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:02:33.960412578 +0000 UTC m=+865.561848915" watchObservedRunningTime="2026-07-31 15:02:33.963237087 +0000 UTC m=+865.564673423" Jul 31 15:02:34.839228 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:34.839179 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:37.712885 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:37.712856 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:37.776353 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:37.776325 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:40.035927 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:40.035897 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:40.135030 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:40.134998 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:41.838962 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:41.838924 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:41.851757 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:41.851721 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:41.994584 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:41.994540 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:02:47.915628 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:02:47.915593 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:13.050367 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.050334 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:13.056342 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.056318 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6"] Jul 31 15:03:13.065368 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:03:13.065333 2582 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 31 15:03:13.065368 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:03:13.065368 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:03:13.065368 ip-10-0-135-143 kubenswrapper[2582]: > containerID="c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50" cmd=["/bin/sleep","15"] Jul 31 15:03:13.065623 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.065388 2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" message="" err=< Jul 31 15:03:13.065623 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:03:13.065623 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:03:13.065623 ip-10-0-135-143 kubenswrapper[2582]: > Jul 31 15:03:13.065623 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:03:13.065400 2582 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 31 15:03:13.065623 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:03:13.065623 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:03:13.065623 ip-10-0-135-143 kubenswrapper[2582]: > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" podUID="fd774962-974f-4b94-867d-2f1381fc74c0" containerName="main" containerID="cri-o://c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50" Jul 31 15:03:13.065623 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.065442 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" podUID="fd774962-974f-4b94-867d-2f1381fc74c0" containerName="main" containerID="cri-o://c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50" gracePeriod=60 Jul 31 15:03:13.146198 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.146164 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:13.818818 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.818793 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:03:13.932839 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.932749 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-model-cache\") pod \"fd774962-974f-4b94-867d-2f1381fc74c0\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " Jul 31 15:03:13.932839 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.932785 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfjqw\" (UniqueName: \"kubernetes.io/projected/fd774962-974f-4b94-867d-2f1381fc74c0-kube-api-access-vfjqw\") pod \"fd774962-974f-4b94-867d-2f1381fc74c0\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " Jul 31 15:03:13.932839 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.932807 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd774962-974f-4b94-867d-2f1381fc74c0-tls-certs\") pod \"fd774962-974f-4b94-867d-2f1381fc74c0\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " Jul 31 15:03:13.933126 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.932858 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-dshm\") pod \"fd774962-974f-4b94-867d-2f1381fc74c0\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " Jul 31 15:03:13.933126 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.932878 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-tmp-dir\") pod \"fd774962-974f-4b94-867d-2f1381fc74c0\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " Jul 31 15:03:13.933126 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.932929 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-home\") pod \"fd774962-974f-4b94-867d-2f1381fc74c0\" (UID: \"fd774962-974f-4b94-867d-2f1381fc74c0\") " Jul 31 15:03:13.933284 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.933119 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd774962-974f-4b94-867d-2f1381fc74c0" (UID: "fd774962-974f-4b94-867d-2f1381fc74c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:13.933284 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.933192 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd774962-974f-4b94-867d-2f1381fc74c0" (UID: "fd774962-974f-4b94-867d-2f1381fc74c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:13.933284 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.933204 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-home" (OuterVolumeSpecName: "home") pod "fd774962-974f-4b94-867d-2f1381fc74c0" (UID: "fd774962-974f-4b94-867d-2f1381fc74c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:13.935142 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.935099 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd774962-974f-4b94-867d-2f1381fc74c0-kube-api-access-vfjqw" (OuterVolumeSpecName: "kube-api-access-vfjqw") pod "fd774962-974f-4b94-867d-2f1381fc74c0" (UID: "fd774962-974f-4b94-867d-2f1381fc74c0"). InnerVolumeSpecName "kube-api-access-vfjqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:03:13.935142 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.935102 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-dshm" (OuterVolumeSpecName: "dshm") pod "fd774962-974f-4b94-867d-2f1381fc74c0" (UID: "fd774962-974f-4b94-867d-2f1381fc74c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:13.935142 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:13.935102 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd774962-974f-4b94-867d-2f1381fc74c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd774962-974f-4b94-867d-2f1381fc74c0" (UID: "fd774962-974f-4b94-867d-2f1381fc74c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:03:14.033926 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.033879 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:14.033926 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.033920 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:14.033926 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.033930 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:14.034154 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.033938 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd774962-974f-4b94-867d-2f1381fc74c0-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:14.034154 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.033947 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfjqw\" (UniqueName: \"kubernetes.io/projected/fd774962-974f-4b94-867d-2f1381fc74c0-kube-api-access-vfjqw\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:14.034154 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.033955 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd774962-974f-4b94-867d-2f1381fc74c0-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:14.155213 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.155182 2582 generic.go:356] "Generic (PLEG): container finished" podID="fd774962-974f-4b94-867d-2f1381fc74c0" containerID="c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50" exitCode=0 Jul 31 15:03:14.155671 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.155227 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" Jul 31 15:03:14.155671 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.155258 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" event={"ID":"fd774962-974f-4b94-867d-2f1381fc74c0","Type":"ContainerDied","Data":"c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50"} Jul 31 15:03:14.155671 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.155315 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6" event={"ID":"fd774962-974f-4b94-867d-2f1381fc74c0","Type":"ContainerDied","Data":"ee76e9496ea575ffeeb6bafb0a97fca9f3be42331683a75d27444200b7052172"} Jul 31 15:03:14.155671 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.155337 2582 scope.go:117] "RemoveContainer" containerID="c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50" Jul 31 15:03:14.165681 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.165662 2582 scope.go:117] "RemoveContainer" containerID="c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50" Jul 31 15:03:14.165965 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:03:14.165949 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50\": container with ID starting with c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50 not found: ID does not exist" containerID="c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50" Jul 31 15:03:14.166019 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.165973 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50"} err="failed to get container status \"c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50\": rpc error: code = NotFound desc = could not find container \"c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50\": container with ID starting with c7aabad5746122e59ceac847d26351dd30c763113dc31ea171baa77638e6de50 not found: ID does not exist" Jul 31 15:03:14.176696 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.176672 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6"] Jul 31 15:03:14.180194 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.180176 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xfxq6"] Jul 31 15:03:14.775525 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:14.775490 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd774962-974f-4b94-867d-2f1381fc74c0" path="/var/lib/kubelet/pods/fd774962-974f-4b94-867d-2f1381fc74c0/volumes" Jul 31 15:03:15.714313 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:15.714277 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:33.026059 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:33.026024 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:33.136428 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:33.136390 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:33.236033 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:33.236001 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:33.373686 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:33.373652 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:33.463553 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:33.463518 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:36.826928 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:36.826889 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:39.380906 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:39.380874 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:47.327013 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:47.326983 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:47.403789 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:47.403756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:52.705718 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.705679 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7"] Jul 31 15:03:52.706176 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.706133 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd774962-974f-4b94-867d-2f1381fc74c0" containerName="main" Jul 31 15:03:52.706176 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.706147 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd774962-974f-4b94-867d-2f1381fc74c0" containerName="main" Jul 31 15:03:52.706356 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.706213 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd774962-974f-4b94-867d-2f1381fc74c0" containerName="main" Jul 31 15:03:52.709327 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.709307 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.712946 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.712921 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-zkd8p\"" Jul 31 15:03:52.712946 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.712938 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\"" Jul 31 15:03:52.713117 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.712921 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\"" Jul 31 15:03:52.713117 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.712978 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\"" Jul 31 15:03:52.719997 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.719976 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7"] Jul 31 15:03:52.739518 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.739461 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:52.873136 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.873105 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.873284 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.873161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.873284 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.873242 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwq4l\" (UniqueName: \"kubernetes.io/projected/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-kube-api-access-bwq4l\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.873382 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.873321 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.873382 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.873355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.873493 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.873455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.974232 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.974160 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.974232 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.974207 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.974395 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.974251 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwq4l\" (UniqueName: \"kubernetes.io/projected/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-kube-api-access-bwq4l\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.974395 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.974301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.974395 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.974327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.974395 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.974378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.974773 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.974750 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.974854 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.974782 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.974854 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.974825 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.976725 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.976697 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.976810 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.976770 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:52.982163 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:52.982144 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwq4l\" (UniqueName: \"kubernetes.io/projected/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-kube-api-access-bwq4l\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:53.021984 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:53.021961 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:53.157117 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:53.157093 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7"] Jul 31 15:03:53.159078 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:03:53.159057 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda312d5d_e6d6_43de_a378_cd2f0d76ba4f.slice/crio-071dbd3e45dedcabe56454776b3317508545d827f29abddf56347f75f98458d9 WatchSource:0}: Error finding container 071dbd3e45dedcabe56454776b3317508545d827f29abddf56347f75f98458d9: Status 404 returned error can't find the container with id 071dbd3e45dedcabe56454776b3317508545d827f29abddf56347f75f98458d9 Jul 31 15:03:53.376302 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:53.376206 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" event={"ID":"da312d5d-e6d6-43de-a378-cd2f0d76ba4f","Type":"ContainerStarted","Data":"35fc9fa14828fc87acb1c48d8cd410578f03b8eb2a868b031274d849075f5719"} Jul 31 15:03:53.376302 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:53.376265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" event={"ID":"da312d5d-e6d6-43de-a378-cd2f0d76ba4f","Type":"ContainerStarted","Data":"071dbd3e45dedcabe56454776b3317508545d827f29abddf56347f75f98458d9"} Jul 31 15:03:53.396460 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:53.396414 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" podStartSLOduration=1.396397241 podStartE2EDuration="1.396397241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:53.39433037 +0000 UTC m=+944.995766711" watchObservedRunningTime="2026-07-31 15:03:53.396397241 +0000 UTC m=+944.997833579" Jul 31 15:03:54.022278 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:54.022246 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:55.136791 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:55.136758 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:55.237715 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:55.237683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:55.339177 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:55.339143 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:55.464904 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:55.464873 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:55.530684 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:55.530656 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:57.129536 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.129500 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7"] Jul 31 15:03:57.140349 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.140319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:57.141249 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:03:57.141221 2582 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 31 15:03:57.141249 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:03:57.141249 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:03:57.141249 ip-10-0-135-143 kubenswrapper[2582]: > containerID="35fc9fa14828fc87acb1c48d8cd410578f03b8eb2a868b031274d849075f5719" cmd=["/bin/sleep","15"] Jul 31 15:03:57.141534 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.141269 2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" message="" err=< Jul 31 15:03:57.141534 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:03:57.141534 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:03:57.141534 ip-10-0-135-143 kubenswrapper[2582]: > Jul 31 15:03:57.141534 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:03:57.141283 2582 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 31 15:03:57.141534 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:03:57.141534 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:03:57.141534 ip-10-0-135-143 kubenswrapper[2582]: > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" podUID="da312d5d-e6d6-43de-a378-cd2f0d76ba4f" containerName="main" containerID="cri-o://35fc9fa14828fc87acb1c48d8cd410578f03b8eb2a868b031274d849075f5719" Jul 31 15:03:57.141534 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.141317 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" podUID="da312d5d-e6d6-43de-a378-cd2f0d76ba4f" containerName="main" containerID="cri-o://35fc9fa14828fc87acb1c48d8cd410578f03b8eb2a868b031274d849075f5719" gracePeriod=60 Jul 31 15:03:57.407652 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.407614 2582 generic.go:356] "Generic (PLEG): container finished" podID="da312d5d-e6d6-43de-a378-cd2f0d76ba4f" containerID="35fc9fa14828fc87acb1c48d8cd410578f03b8eb2a868b031274d849075f5719" exitCode=0 Jul 31 15:03:57.407833 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.407682 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" event={"ID":"da312d5d-e6d6-43de-a378-cd2f0d76ba4f","Type":"ContainerDied","Data":"35fc9fa14828fc87acb1c48d8cd410578f03b8eb2a868b031274d849075f5719"} Jul 31 15:03:57.491249 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.491221 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:57.617009 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.616977 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tmp-dir\") pod \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " Jul 31 15:03:57.617175 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.617016 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-home\") pod \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " Jul 31 15:03:57.617175 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.617076 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-dshm\") pod \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " Jul 31 15:03:57.617175 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.617125 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwq4l\" (UniqueName: \"kubernetes.io/projected/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-kube-api-access-bwq4l\") pod \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " Jul 31 15:03:57.617175 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.617165 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tls-certs\") pod \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " Jul 31 15:03:57.617391 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.617208 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-model-cache\") pod \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\" (UID: \"da312d5d-e6d6-43de-a378-cd2f0d76ba4f\") " Jul 31 15:03:57.617391 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.617277 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da312d5d-e6d6-43de-a378-cd2f0d76ba4f" (UID: "da312d5d-e6d6-43de-a378-cd2f0d76ba4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:57.617391 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.617296 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-home" (OuterVolumeSpecName: "home") pod "da312d5d-e6d6-43de-a378-cd2f0d76ba4f" (UID: "da312d5d-e6d6-43de-a378-cd2f0d76ba4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:57.617542 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.617500 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "da312d5d-e6d6-43de-a378-cd2f0d76ba4f" (UID: "da312d5d-e6d6-43de-a378-cd2f0d76ba4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:57.617542 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.617504 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:57.617542 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.617531 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:57.619318 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.619290 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-dshm" (OuterVolumeSpecName: "dshm") pod "da312d5d-e6d6-43de-a378-cd2f0d76ba4f" (UID: "da312d5d-e6d6-43de-a378-cd2f0d76ba4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:57.619418 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.619340 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da312d5d-e6d6-43de-a378-cd2f0d76ba4f" (UID: "da312d5d-e6d6-43de-a378-cd2f0d76ba4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:03:57.619418 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.619352 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-kube-api-access-bwq4l" (OuterVolumeSpecName: "kube-api-access-bwq4l") pod "da312d5d-e6d6-43de-a378-cd2f0d76ba4f" (UID: "da312d5d-e6d6-43de-a378-cd2f0d76ba4f"). InnerVolumeSpecName "kube-api-access-bwq4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:03:57.718367 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.718299 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwq4l\" (UniqueName: \"kubernetes.io/projected/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-kube-api-access-bwq4l\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:57.718367 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.718326 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:57.718367 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.718339 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:57.718367 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:57.718347 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da312d5d-e6d6-43de-a378-cd2f0d76ba4f-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:03:58.415354 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:58.415326 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" Jul 31 15:03:58.415754 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:58.415325 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7" event={"ID":"da312d5d-e6d6-43de-a378-cd2f0d76ba4f","Type":"ContainerDied","Data":"071dbd3e45dedcabe56454776b3317508545d827f29abddf56347f75f98458d9"} Jul 31 15:03:58.415754 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:58.415467 2582 scope.go:117] "RemoveContainer" containerID="35fc9fa14828fc87acb1c48d8cd410578f03b8eb2a868b031274d849075f5719" Jul 31 15:03:58.437822 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:58.437794 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7"] Jul 31 15:03:58.441846 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:58.441824 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-4tqh7"] Jul 31 15:03:58.675335 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:58.675267 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:58.775568 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:03:58.775528 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da312d5d-e6d6-43de-a378-cd2f0d76ba4f" path="/var/lib/kubelet/pods/da312d5d-e6d6-43de-a378-cd2f0d76ba4f/volumes" Jul 31 15:04:00.214328 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:00.214296 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:07.778018 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:07.777983 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:07.835238 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:07.835203 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:08.096606 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.096498 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz"] Jul 31 15:04:08.097074 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.097041 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da312d5d-e6d6-43de-a378-cd2f0d76ba4f" containerName="main" Jul 31 15:04:08.097074 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.097065 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="da312d5d-e6d6-43de-a378-cd2f0d76ba4f" containerName="main" Jul 31 15:04:08.097272 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.097176 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="da312d5d-e6d6-43de-a378-cd2f0d76ba4f" containerName="main" Jul 31 15:04:08.100168 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.100149 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.102684 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.102666 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\"" Jul 31 15:04:08.103384 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.103362 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\"" Jul 31 15:04:08.103499 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.103448 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-kgqwq\"" Jul 31 15:04:08.103552 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.103447 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\"" Jul 31 15:04:08.114171 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.114151 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz"] Jul 31 15:04:08.122323 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.122301 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:08.202761 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.202727 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.202916 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.202762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.202916 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.202794 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.202916 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.202876 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34eeea0-0278-4005-8d85-d79930ae6eb3-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.203027 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.202941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.203027 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.202979 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdp7\" (UniqueName: \"kubernetes.io/projected/e34eeea0-0278-4005-8d85-d79930ae6eb3-kube-api-access-sfdp7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.304001 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.303970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdp7\" (UniqueName: \"kubernetes.io/projected/e34eeea0-0278-4005-8d85-d79930ae6eb3-kube-api-access-sfdp7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.304191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.304014 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.304191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.304034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.304191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.304052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.304191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.304096 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34eeea0-0278-4005-8d85-d79930ae6eb3-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.304191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.304138 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.304520 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.304440 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.304520 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.304454 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.304638 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.304534 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.306389 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.306367 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.306633 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.306614 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34eeea0-0278-4005-8d85-d79930ae6eb3-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.311658 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.311630 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdp7\" (UniqueName: \"kubernetes.io/projected/e34eeea0-0278-4005-8d85-d79930ae6eb3-kube-api-access-sfdp7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.412265 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.412240 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:08.549055 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:08.549033 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz"] Jul 31 15:04:08.550931 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:04:08.550905 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34eeea0_0278_4005_8d85_d79930ae6eb3.slice/crio-9c3227cb7fb9bb8d9e7e3be71c1d9b704c2aa0de1675e518a02d09f5832b0d3b WatchSource:0}: Error finding container 9c3227cb7fb9bb8d9e7e3be71c1d9b704c2aa0de1675e518a02d09f5832b0d3b: Status 404 returned error can't find the container with id 9c3227cb7fb9bb8d9e7e3be71c1d9b704c2aa0de1675e518a02d09f5832b0d3b Jul 31 15:04:09.477800 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:09.477765 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" event={"ID":"e34eeea0-0278-4005-8d85-d79930ae6eb3","Type":"ContainerStarted","Data":"5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3"} Jul 31 15:04:09.478231 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:09.477822 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" event={"ID":"e34eeea0-0278-4005-8d85-d79930ae6eb3","Type":"ContainerStarted","Data":"9c3227cb7fb9bb8d9e7e3be71c1d9b704c2aa0de1675e518a02d09f5832b0d3b"} Jul 31 15:04:09.496853 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:09.496803 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" podStartSLOduration=1.4967878780000001 podStartE2EDuration="1.496787878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:09.494604918 +0000 UTC m=+961.096041258" watchObservedRunningTime="2026-07-31 15:04:09.496787878 +0000 UTC m=+961.098224247" Jul 31 15:04:10.127391 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:10.127358 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:10.175240 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:10.175206 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh"] Jul 31 15:04:10.198565 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:10.198535 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:10.262184 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:10.262152 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:10.412898 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:10.412868 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:10.587570 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:10.587524 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" podUID="fc56912c-23c7-47da-82e0-8801c843203a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:04:10.995313 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:10.995283 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz"] Jul 31 15:04:11.017246 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:11.017218 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:11.531127 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:04:11.531097 2582 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found Jul 31 15:04:11.531326 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:04:11.531186 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e34eeea0-0278-4005-8d85-d79930ae6eb3-tls-certs podName:e34eeea0-0278-4005-8d85-d79930ae6eb3 nodeName:}" failed. No retries permitted until 2026-07-31 15:04:12.03116496 +0000 UTC m=+963.632601278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e34eeea0-0278-4005-8d85-d79930ae6eb3-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" (UID: "e34eeea0-0278-4005-8d85-d79930ae6eb3") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found Jul 31 15:04:12.027287 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.027257 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:12.036633 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:04:12.036601 2582 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found Jul 31 15:04:12.036760 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:04:12.036671 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e34eeea0-0278-4005-8d85-d79930ae6eb3-tls-certs podName:e34eeea0-0278-4005-8d85-d79930ae6eb3 nodeName:}" failed. No retries permitted until 2026-07-31 15:04:13.036654872 +0000 UTC m=+964.638091186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e34eeea0-0278-4005-8d85-d79930ae6eb3-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" (UID: "e34eeea0-0278-4005-8d85-d79930ae6eb3") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found Jul 31 15:04:12.501895 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:04:12.501854 2582 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 31 15:04:12.501895 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:04:12.501895 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:04:12.501895 ip-10-0-135-143 kubenswrapper[2582]: > containerID="5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3" cmd=["/bin/sleep","15"] Jul 31 15:04:12.501895 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.501896 2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" message="" err=< Jul 31 15:04:12.501895 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:04:12.501895 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:04:12.501895 ip-10-0-135-143 kubenswrapper[2582]: > Jul 31 15:04:12.502317 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:04:12.501910 2582 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 31 15:04:12.502317 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:04:12.502317 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:04:12.502317 ip-10-0-135-143 kubenswrapper[2582]: > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" podUID="e34eeea0-0278-4005-8d85-d79930ae6eb3" containerName="main" containerID="cri-o://5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3" Jul 31 15:04:12.502317 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.501952 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" podUID="e34eeea0-0278-4005-8d85-d79930ae6eb3" containerName="main" containerID="cri-o://5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3" gracePeriod=60 Jul 31 15:04:12.756926 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.756874 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:12.844840 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.844811 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34eeea0-0278-4005-8d85-d79930ae6eb3-tls-certs\") pod \"e34eeea0-0278-4005-8d85-d79930ae6eb3\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " Jul 31 15:04:12.844981 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.844856 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfdp7\" (UniqueName: \"kubernetes.io/projected/e34eeea0-0278-4005-8d85-d79930ae6eb3-kube-api-access-sfdp7\") pod \"e34eeea0-0278-4005-8d85-d79930ae6eb3\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " Jul 31 15:04:12.844981 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.844904 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-home\") pod \"e34eeea0-0278-4005-8d85-d79930ae6eb3\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " Jul 31 15:04:12.844981 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.844920 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-tmp-dir\") pod \"e34eeea0-0278-4005-8d85-d79930ae6eb3\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " Jul 31 15:04:12.844981 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.844949 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-dshm\") pod \"e34eeea0-0278-4005-8d85-d79930ae6eb3\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " Jul 31 15:04:12.845211 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.844988 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-model-cache\") pod \"e34eeea0-0278-4005-8d85-d79930ae6eb3\" (UID: \"e34eeea0-0278-4005-8d85-d79930ae6eb3\") " Jul 31 15:04:12.845211 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.845184 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-home" (OuterVolumeSpecName: "home") pod "e34eeea0-0278-4005-8d85-d79930ae6eb3" (UID: "e34eeea0-0278-4005-8d85-d79930ae6eb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:12.845323 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.845203 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e34eeea0-0278-4005-8d85-d79930ae6eb3" (UID: "e34eeea0-0278-4005-8d85-d79930ae6eb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:12.845323 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.845279 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:04:12.845323 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.845310 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "e34eeea0-0278-4005-8d85-d79930ae6eb3" (UID: "e34eeea0-0278-4005-8d85-d79930ae6eb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:12.847109 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.847089 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-dshm" (OuterVolumeSpecName: "dshm") pod "e34eeea0-0278-4005-8d85-d79930ae6eb3" (UID: "e34eeea0-0278-4005-8d85-d79930ae6eb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:12.847109 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.847104 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34eeea0-0278-4005-8d85-d79930ae6eb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e34eeea0-0278-4005-8d85-d79930ae6eb3" (UID: "e34eeea0-0278-4005-8d85-d79930ae6eb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:04:12.847254 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.847128 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34eeea0-0278-4005-8d85-d79930ae6eb3-kube-api-access-sfdp7" (OuterVolumeSpecName: "kube-api-access-sfdp7") pod "e34eeea0-0278-4005-8d85-d79930ae6eb3" (UID: "e34eeea0-0278-4005-8d85-d79930ae6eb3"). InnerVolumeSpecName "kube-api-access-sfdp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:12.945656 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.945635 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34eeea0-0278-4005-8d85-d79930ae6eb3-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:04:12.945656 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.945655 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfdp7\" (UniqueName: \"kubernetes.io/projected/e34eeea0-0278-4005-8d85-d79930ae6eb3-kube-api-access-sfdp7\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:04:12.945797 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.945665 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:04:12.945797 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.945675 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:04:12.945797 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:12.945685 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34eeea0-0278-4005-8d85-d79930ae6eb3-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:04:13.499855 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:13.499822 2582 generic.go:356] "Generic (PLEG): container finished" podID="e34eeea0-0278-4005-8d85-d79930ae6eb3" containerID="5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3" exitCode=0 Jul 31 15:04:13.500233 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:13.499869 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" Jul 31 15:04:13.500233 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:13.499913 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" event={"ID":"e34eeea0-0278-4005-8d85-d79930ae6eb3","Type":"ContainerDied","Data":"5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3"} Jul 31 15:04:13.500233 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:13.499968 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz" event={"ID":"e34eeea0-0278-4005-8d85-d79930ae6eb3","Type":"ContainerDied","Data":"9c3227cb7fb9bb8d9e7e3be71c1d9b704c2aa0de1675e518a02d09f5832b0d3b"} Jul 31 15:04:13.500233 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:13.499990 2582 scope.go:117] "RemoveContainer" containerID="5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3" Jul 31 15:04:13.509462 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:13.509281 2582 scope.go:117] "RemoveContainer" containerID="5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3" Jul 31 15:04:13.509537 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:04:13.509526 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3\": container with ID starting with 5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3 not found: ID does not exist" containerID="5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3" Jul 31 15:04:13.509599 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:13.509550 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3"} err="failed to get container status \"5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3\": rpc error: code = NotFound desc = could not find container \"5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3\": container with ID starting with 5cd08700f965fcebe5cae5c1efad75f077b1a209ede94ba14e4b6c9f475118b3 not found: ID does not exist" Jul 31 15:04:13.521449 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:13.521423 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz"] Jul 31 15:04:13.526099 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:13.526080 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf8xltz"] Jul 31 15:04:14.776101 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:14.776069 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e34eeea0-0278-4005-8d85-d79930ae6eb3" path="/var/lib/kubelet/pods/e34eeea0-0278-4005-8d85-d79930ae6eb3/volumes" Jul 31 15:04:15.219682 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:15.219647 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:16.069362 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:16.069326 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:16.125848 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:16.125821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:20.556328 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.556298 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:20.587657 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.587630 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" podUID="fc56912c-23c7-47da-82e0-8801c843203a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:04:20.950763 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.950737 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:20.985087 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.985059 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp"] Jul 31 15:04:20.985457 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.985443 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e34eeea0-0278-4005-8d85-d79930ae6eb3" containerName="main" Jul 31 15:04:20.985457 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.985456 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e34eeea0-0278-4005-8d85-d79930ae6eb3" containerName="main" Jul 31 15:04:20.985606 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.985511 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e34eeea0-0278-4005-8d85-d79930ae6eb3" containerName="main" Jul 31 15:04:20.989907 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.989888 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:04:20.993773 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.993748 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-52d25\"" Jul 31 15:04:20.993885 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.993802 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\"" Jul 31 15:04:20.993885 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.993828 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-2nxrd\"" Jul 31 15:04:20.993885 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.993753 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\"" Jul 31 15:04:20.993885 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.993803 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\"" Jul 31 15:04:20.998728 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:20.998707 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp"] Jul 31 15:04:21.051500 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.051465 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:21.113341 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.113307 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm45l\" (UniqueName: \"kubernetes.io/projected/198c1078-246a-4a5a-9fc5-f5e87bda1670-kube-api-access-jm45l\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp\" (UID: \"198c1078-246a-4a5a-9fc5-f5e87bda1670\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:04:21.113480 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.113351 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198c1078-246a-4a5a-9fc5-f5e87bda1670-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp\" (UID: \"198c1078-246a-4a5a-9fc5-f5e87bda1670\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:04:21.214262 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.214194 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm45l\" (UniqueName: \"kubernetes.io/projected/198c1078-246a-4a5a-9fc5-f5e87bda1670-kube-api-access-jm45l\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp\" (UID: \"198c1078-246a-4a5a-9fc5-f5e87bda1670\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:04:21.214262 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.214237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198c1078-246a-4a5a-9fc5-f5e87bda1670-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp\" (UID: \"198c1078-246a-4a5a-9fc5-f5e87bda1670\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:04:21.216765 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.216744 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198c1078-246a-4a5a-9fc5-f5e87bda1670-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp\" (UID: \"198c1078-246a-4a5a-9fc5-f5e87bda1670\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:04:21.225831 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.225804 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm45l\" (UniqueName: \"kubernetes.io/projected/198c1078-246a-4a5a-9fc5-f5e87bda1670-kube-api-access-jm45l\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp\" (UID: \"198c1078-246a-4a5a-9fc5-f5e87bda1670\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:04:21.301280 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.301252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:04:21.440687 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.440616 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp"] Jul 31 15:04:21.443521 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:04:21.443489 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198c1078_246a_4a5a_9fc5_f5e87bda1670.slice/crio-292c3ca0e58f561ce81753eb812ce2dd17f01c072e4740061fe8aa3c944d1a8b WatchSource:0}: Error finding container 292c3ca0e58f561ce81753eb812ce2dd17f01c072e4740061fe8aa3c944d1a8b: Status 404 returned error can't find the container with id 292c3ca0e58f561ce81753eb812ce2dd17f01c072e4740061fe8aa3c944d1a8b Jul 31 15:04:21.546379 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.546346 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" event={"ID":"198c1078-246a-4a5a-9fc5-f5e87bda1670","Type":"ContainerStarted","Data":"d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928"} Jul 31 15:04:21.546513 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.546394 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" event={"ID":"198c1078-246a-4a5a-9fc5-f5e87bda1670","Type":"ContainerStarted","Data":"292c3ca0e58f561ce81753eb812ce2dd17f01c072e4740061fe8aa3c944d1a8b"} Jul 31 15:04:21.546513 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.546472 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:04:21.572610 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:21.572549 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" podStartSLOduration=1.572533114 podStartE2EDuration="1.572533114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:21.571443988 +0000 UTC m=+973.172880346" watchObservedRunningTime="2026-07-31 15:04:21.572533114 +0000 UTC m=+973.173969476" Jul 31 15:04:22.104904 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:22.104867 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:22.553538 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:22.553495 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:22.616098 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:22.616069 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:25.194730 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.194680 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" podUID="fc56912c-23c7-47da-82e0-8801c843203a" containerName="main" containerID="cri-o://c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517" gracePeriod=45 Jul 31 15:04:25.445519 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.445467 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:04:25.531410 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.531380 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:25.549916 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.549890 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc56912c-23c7-47da-82e0-8801c843203a-tls-certs\") pod \"fc56912c-23c7-47da-82e0-8801c843203a\" (UID: \"fc56912c-23c7-47da-82e0-8801c843203a\") " Jul 31 15:04:25.550031 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.549963 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkjmr\" (UniqueName: \"kubernetes.io/projected/fc56912c-23c7-47da-82e0-8801c843203a-kube-api-access-zkjmr\") pod \"fc56912c-23c7-47da-82e0-8801c843203a\" (UID: \"fc56912c-23c7-47da-82e0-8801c843203a\") " Jul 31 15:04:25.551952 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.551926 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc56912c-23c7-47da-82e0-8801c843203a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc56912c-23c7-47da-82e0-8801c843203a" (UID: "fc56912c-23c7-47da-82e0-8801c843203a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:04:25.552046 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.551952 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc56912c-23c7-47da-82e0-8801c843203a-kube-api-access-zkjmr" (OuterVolumeSpecName: "kube-api-access-zkjmr") pod "fc56912c-23c7-47da-82e0-8801c843203a" (UID: "fc56912c-23c7-47da-82e0-8801c843203a"). InnerVolumeSpecName "kube-api-access-zkjmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:25.577846 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.577810 2582 generic.go:356] "Generic (PLEG): container finished" podID="fc56912c-23c7-47da-82e0-8801c843203a" containerID="c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517" exitCode=0 Jul 31 15:04:25.577986 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.577877 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" Jul 31 15:04:25.577986 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.577895 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" event={"ID":"fc56912c-23c7-47da-82e0-8801c843203a","Type":"ContainerDied","Data":"c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517"} Jul 31 15:04:25.577986 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.577980 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh" event={"ID":"fc56912c-23c7-47da-82e0-8801c843203a","Type":"ContainerDied","Data":"321bafd865f7a82d55ebbb28ca8be5b5867323885b58fec5adc7083788621890"} Jul 31 15:04:25.578141 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.578008 2582 scope.go:117] "RemoveContainer" containerID="c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517" Jul 31 15:04:25.587546 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.587528 2582 scope.go:117] "RemoveContainer" containerID="c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517" Jul 31 15:04:25.587779 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:04:25.587760 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517\": container with ID starting with c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517 not found: ID does not exist" containerID="c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517" Jul 31 15:04:25.587840 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.587788 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517"} err="failed to get container status \"c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517\": rpc error: code = NotFound desc = could not find container \"c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517\": container with ID starting with c082b92cd84c44b7dd315471ffd7b854cf85f6717320f74cba21e6808bf92517 not found: ID does not exist" Jul 31 15:04:25.599761 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.599738 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh"] Jul 31 15:04:25.606104 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.606086 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche58wfh"] Jul 31 15:04:25.650481 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.650459 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc56912c-23c7-47da-82e0-8801c843203a-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:04:25.650481 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:25.650481 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkjmr\" (UniqueName: \"kubernetes.io/projected/fc56912c-23c7-47da-82e0-8801c843203a-kube-api-access-zkjmr\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:04:26.781435 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:26.781401 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc56912c-23c7-47da-82e0-8801c843203a" path="/var/lib/kubelet/pods/fc56912c-23c7-47da-82e0-8801c843203a/volumes" Jul 31 15:04:30.439767 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:30.439715 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:37.210489 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:37.210451 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:37.311924 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:37.311892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:46.222518 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:46.222477 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:46.957200 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:46.957159 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:47.088661 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:47.088635 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:52.557148 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:52.557117 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:04:53.101324 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:53.101292 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:56.333148 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:56.333113 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:56.597216 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:56.597112 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:59.730200 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:59.730165 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:59.805869 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:04:59.805832 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:26.901963 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:05:26.901929 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:39.019870 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:05:39.019830 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:44.736222 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:05:44.736172 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:44.811977 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:05:44.811947 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:04.756689 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:04.756616 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:04.852853 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:04.852821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:11.234883 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:11.234804 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:13.499107 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:13.499068 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:13.763347 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:13.763262 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:16.983376 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:16.983322 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:17.071946 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:17.071908 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:18.020352 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:18.020317 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:25.394135 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:25.394098 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp"] Jul 31 15:06:25.395152 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:25.395105 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:25.460692 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:25.460661 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:32.557340 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:32.557299 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" podUID="198c1078-246a-4a5a-9fc5-f5e87bda1670" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:06:40.413376 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:40.413315 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" podUID="198c1078-246a-4a5a-9fc5-f5e87bda1670" containerName="main" containerID="cri-o://d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928" gracePeriod=45 Jul 31 15:06:40.673010 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:40.672956 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:06:40.786889 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:40.786860 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198c1078-246a-4a5a-9fc5-f5e87bda1670-tls-certs\") pod \"198c1078-246a-4a5a-9fc5-f5e87bda1670\" (UID: \"198c1078-246a-4a5a-9fc5-f5e87bda1670\") " Jul 31 15:06:40.787037 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:40.786904 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm45l\" (UniqueName: \"kubernetes.io/projected/198c1078-246a-4a5a-9fc5-f5e87bda1670-kube-api-access-jm45l\") pod \"198c1078-246a-4a5a-9fc5-f5e87bda1670\" (UID: \"198c1078-246a-4a5a-9fc5-f5e87bda1670\") " Jul 31 15:06:40.788945 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:40.788917 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198c1078-246a-4a5a-9fc5-f5e87bda1670-kube-api-access-jm45l" (OuterVolumeSpecName: "kube-api-access-jm45l") pod "198c1078-246a-4a5a-9fc5-f5e87bda1670" (UID: "198c1078-246a-4a5a-9fc5-f5e87bda1670"). InnerVolumeSpecName "kube-api-access-jm45l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:06:40.789053 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:40.788943 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198c1078-246a-4a5a-9fc5-f5e87bda1670-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "198c1078-246a-4a5a-9fc5-f5e87bda1670" (UID: "198c1078-246a-4a5a-9fc5-f5e87bda1670"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:06:40.888001 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:40.887975 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198c1078-246a-4a5a-9fc5-f5e87bda1670-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:06:40.888001 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:40.887999 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm45l\" (UniqueName: \"kubernetes.io/projected/198c1078-246a-4a5a-9fc5-f5e87bda1670-kube-api-access-jm45l\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:06:41.297845 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:41.297813 2582 generic.go:356] "Generic (PLEG): container finished" podID="198c1078-246a-4a5a-9fc5-f5e87bda1670" containerID="d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928" exitCode=0 Jul 31 15:06:41.298012 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:41.297856 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" Jul 31 15:06:41.298012 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:41.297896 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" event={"ID":"198c1078-246a-4a5a-9fc5-f5e87bda1670","Type":"ContainerDied","Data":"d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928"} Jul 31 15:06:41.298012 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:41.297950 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp" event={"ID":"198c1078-246a-4a5a-9fc5-f5e87bda1670","Type":"ContainerDied","Data":"292c3ca0e58f561ce81753eb812ce2dd17f01c072e4740061fe8aa3c944d1a8b"} Jul 31 15:06:41.298012 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:41.297972 2582 scope.go:117] "RemoveContainer" containerID="d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928" Jul 31 15:06:41.307082 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:41.307049 2582 scope.go:117] "RemoveContainer" containerID="d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928" Jul 31 15:06:41.307324 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:06:41.307306 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928\": container with ID starting with d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928 not found: ID does not exist" containerID="d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928" Jul 31 15:06:41.307388 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:41.307338 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928"} err="failed to get container status \"d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928\": rpc error: code = NotFound desc = could not find container \"d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928\": container with ID starting with d92c8f1e560ec188749253719b6ace76c09823777a1729b8e4720fedbb731928 not found: ID does not exist" Jul 31 15:06:41.319356 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:41.319335 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp"] Jul 31 15:06:41.325623 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:41.325594 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-zr6tp"] Jul 31 15:06:42.775485 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:42.775449 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="198c1078-246a-4a5a-9fc5-f5e87bda1670" path="/var/lib/kubelet/pods/198c1078-246a-4a5a-9fc5-f5e87bda1670/volumes" Jul 31 15:06:45.440299 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:45.440266 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:45.528381 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:45.528354 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:52.071898 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:52.071863 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:55.449630 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.449600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:55.680023 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.679983 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl"] Jul 31 15:06:55.680880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.680858 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc56912c-23c7-47da-82e0-8801c843203a" containerName="main" Jul 31 15:06:55.680880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.680880 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc56912c-23c7-47da-82e0-8801c843203a" containerName="main" Jul 31 15:06:55.681071 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.680899 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="198c1078-246a-4a5a-9fc5-f5e87bda1670" containerName="main" Jul 31 15:06:55.681071 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.680907 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="198c1078-246a-4a5a-9fc5-f5e87bda1670" containerName="main" Jul 31 15:06:55.681071 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.680991 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc56912c-23c7-47da-82e0-8801c843203a" containerName="main" Jul 31 15:06:55.681071 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.681001 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="198c1078-246a-4a5a-9fc5-f5e87bda1670" containerName="main" Jul 31 15:06:55.685756 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.685730 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:06:55.686201 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.686180 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl"] Jul 31 15:06:55.688171 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.688151 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\"" Jul 31 15:06:55.689049 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.689022 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\"" Jul 31 15:06:55.689152 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.689082 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\"" Jul 31 15:06:55.689152 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.689092 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-vjbls\"" Jul 31 15:06:55.689330 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.689023 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-xrdkg\"" Jul 31 15:06:55.701370 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.701325 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:55.704367 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.704350 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgsh\" (UniqueName: \"kubernetes.io/projected/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-kube-api-access-rmgsh\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl\" (UID: \"ba26204c-c84b-4b1d-94cd-e4be467fdb5b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:06:55.704439 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.704385 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl\" (UID: \"ba26204c-c84b-4b1d-94cd-e4be467fdb5b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:06:55.805622 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.805590 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl\" (UID: \"ba26204c-c84b-4b1d-94cd-e4be467fdb5b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:06:55.805751 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.805702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgsh\" (UniqueName: \"kubernetes.io/projected/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-kube-api-access-rmgsh\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl\" (UID: \"ba26204c-c84b-4b1d-94cd-e4be467fdb5b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:06:55.808023 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.808001 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl\" (UID: \"ba26204c-c84b-4b1d-94cd-e4be467fdb5b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:06:55.814652 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.814626 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgsh\" (UniqueName: \"kubernetes.io/projected/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-kube-api-access-rmgsh\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl\" (UID: \"ba26204c-c84b-4b1d-94cd-e4be467fdb5b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:06:55.996267 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:55.996187 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:06:56.126093 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:56.126070 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl"] Jul 31 15:06:56.128156 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:06:56.128130 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba26204c_c84b_4b1d_94cd_e4be467fdb5b.slice/crio-b3628d39cf5c7b385e60793dc1c4b4e43be7c4ac7b192770ea688e3c2227deef WatchSource:0}: Error finding container b3628d39cf5c7b385e60793dc1c4b4e43be7c4ac7b192770ea688e3c2227deef: Status 404 returned error can't find the container with id b3628d39cf5c7b385e60793dc1c4b4e43be7c4ac7b192770ea688e3c2227deef Jul 31 15:06:56.382725 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:56.382690 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" event={"ID":"ba26204c-c84b-4b1d-94cd-e4be467fdb5b","Type":"ContainerStarted","Data":"e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f"} Jul 31 15:06:56.382880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:56.382750 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" event={"ID":"ba26204c-c84b-4b1d-94cd-e4be467fdb5b","Type":"ContainerStarted","Data":"b3628d39cf5c7b385e60793dc1c4b4e43be7c4ac7b192770ea688e3c2227deef"} Jul 31 15:06:56.382880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:56.382788 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:06:56.404300 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:56.404257 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" podStartSLOduration=1.404239981 podStartE2EDuration="1.404239981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:06:56.403331028 +0000 UTC m=+1128.004767364" watchObservedRunningTime="2026-07-31 15:06:56.404239981 +0000 UTC m=+1128.005676319" Jul 31 15:06:57.927024 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:57.926954 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:58.021596 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:58.021568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:58.430023 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:58.429995 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:58.493416 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:06:58.493386 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:06.874240 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:06.874204 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:17.844830 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:17.844787 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:17.941802 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:17.941769 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:21.139508 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:21.139475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:21.459067 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:21.458974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:24.948461 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:24.948419 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:27.391849 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:27.391823 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:07:30.815039 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:30.814940 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:30.877185 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:30.877135 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:32.310514 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:32.310484 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:54.419015 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:54.418984 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:54.518627 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:07:54.518572 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:14.561745 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:14.561698 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:14.662196 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:14.662129 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:20.612897 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:20.612856 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:23.498793 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.498756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:23.760199 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.760120 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx"] Jul 31 15:08:23.763841 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.763823 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:08:23.767209 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.767178 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\"" Jul 31 15:08:23.767209 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.767204 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\"" Jul 31 15:08:23.767402 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.767222 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-kc9xd\"" Jul 31 15:08:23.767402 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.767276 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\"" Jul 31 15:08:23.767402 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.767327 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hmprv\"" Jul 31 15:08:23.773373 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.773351 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx"] Jul 31 15:08:23.781343 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.781321 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:23.816876 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.816852 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a081d6-1e2a-4777-b976-4f6a6dfe6016-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx\" (UID: \"49a081d6-1e2a-4777-b976-4f6a6dfe6016\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:08:23.817016 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.816951 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jptn\" (UniqueName: \"kubernetes.io/projected/49a081d6-1e2a-4777-b976-4f6a6dfe6016-kube-api-access-4jptn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx\" (UID: \"49a081d6-1e2a-4777-b976-4f6a6dfe6016\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:08:23.917878 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.917842 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a081d6-1e2a-4777-b976-4f6a6dfe6016-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx\" (UID: \"49a081d6-1e2a-4777-b976-4f6a6dfe6016\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:08:23.918037 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.917932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jptn\" (UniqueName: \"kubernetes.io/projected/49a081d6-1e2a-4777-b976-4f6a6dfe6016-kube-api-access-4jptn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx\" (UID: \"49a081d6-1e2a-4777-b976-4f6a6dfe6016\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:08:23.920398 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.920369 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a081d6-1e2a-4777-b976-4f6a6dfe6016-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx\" (UID: \"49a081d6-1e2a-4777-b976-4f6a6dfe6016\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:08:23.926009 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:23.925980 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jptn\" (UniqueName: \"kubernetes.io/projected/49a081d6-1e2a-4777-b976-4f6a6dfe6016-kube-api-access-4jptn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx\" (UID: \"49a081d6-1e2a-4777-b976-4f6a6dfe6016\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:08:24.074891 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:24.074826 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:08:24.415388 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:24.415365 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx"] Jul 31 15:08:24.417066 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:08:24.417037 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a081d6_1e2a_4777_b976_4f6a6dfe6016.slice/crio-2130451664ad31876b2487d5aa83cc2c0a99d186adab01d60f1bfccc9c51aac2 WatchSource:0}: Error finding container 2130451664ad31876b2487d5aa83cc2c0a99d186adab01d60f1bfccc9c51aac2: Status 404 returned error can't find the container with id 2130451664ad31876b2487d5aa83cc2c0a99d186adab01d60f1bfccc9c51aac2 Jul 31 15:08:24.850605 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:24.850509 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" event={"ID":"49a081d6-1e2a-4777-b976-4f6a6dfe6016","Type":"ContainerStarted","Data":"379dd90d9f250cacd45fb019d331c65906cd038521b9ea3fef8ead736dcc14eb"} Jul 31 15:08:24.850605 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:24.850582 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" event={"ID":"49a081d6-1e2a-4777-b976-4f6a6dfe6016","Type":"ContainerStarted","Data":"2130451664ad31876b2487d5aa83cc2c0a99d186adab01d60f1bfccc9c51aac2"} Jul 31 15:08:24.851027 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:24.850636 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:08:24.880079 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:24.880034 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" podStartSLOduration=1.880018458 podStartE2EDuration="1.880018458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:08:24.877628808 +0000 UTC m=+1216.479065147" watchObservedRunningTime="2026-07-31 15:08:24.880018458 +0000 UTC m=+1216.481454795" Jul 31 15:08:26.817708 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:26.817676 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:26.970692 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:26.970662 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:29.578230 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:29.578198 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:29.688130 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:29.688100 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:32.480411 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:32.480377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:55.858252 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:55.858219 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" podUID="49a081d6-1e2a-4777-b976-4f6a6dfe6016" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:08:57.565462 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:57.565430 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:08:57.575115 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:57.575089 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl"] Jul 31 15:08:58.852732 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:58.852697 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx"] Jul 31 15:08:58.854604 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:58.854551 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" podUID="49a081d6-1e2a-4777-b976-4f6a6dfe6016" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:08:59.246456 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:08:59.246427 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:07.391988 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:07.391898 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" podUID="ba26204c-c84b-4b1d-94cd-e4be467fdb5b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:09:08.854478 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:08.854440 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" podUID="49a081d6-1e2a-4777-b976-4f6a6dfe6016" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:09:12.594049 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:12.594004 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" podUID="ba26204c-c84b-4b1d-94cd-e4be467fdb5b" containerName="main" containerID="cri-o://e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f" gracePeriod=45 Jul 31 15:09:12.843274 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:12.843251 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:09:12.927573 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:12.927537 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-tls-certs\") pod \"ba26204c-c84b-4b1d-94cd-e4be467fdb5b\" (UID: \"ba26204c-c84b-4b1d-94cd-e4be467fdb5b\") " Jul 31 15:09:12.927748 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:12.927644 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmgsh\" (UniqueName: \"kubernetes.io/projected/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-kube-api-access-rmgsh\") pod \"ba26204c-c84b-4b1d-94cd-e4be467fdb5b\" (UID: \"ba26204c-c84b-4b1d-94cd-e4be467fdb5b\") " Jul 31 15:09:12.929725 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:12.929700 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba26204c-c84b-4b1d-94cd-e4be467fdb5b" (UID: "ba26204c-c84b-4b1d-94cd-e4be467fdb5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:09:12.929725 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:12.929709 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-kube-api-access-rmgsh" (OuterVolumeSpecName: "kube-api-access-rmgsh") pod "ba26204c-c84b-4b1d-94cd-e4be467fdb5b" (UID: "ba26204c-c84b-4b1d-94cd-e4be467fdb5b"). InnerVolumeSpecName "kube-api-access-rmgsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:09:13.028757 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.028726 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmgsh\" (UniqueName: \"kubernetes.io/projected/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-kube-api-access-rmgsh\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:09:13.028757 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.028749 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba26204c-c84b-4b1d-94cd-e4be467fdb5b-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:09:13.103606 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.103578 2582 generic.go:356] "Generic (PLEG): container finished" podID="ba26204c-c84b-4b1d-94cd-e4be467fdb5b" containerID="e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f" exitCode=0 Jul 31 15:09:13.103727 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.103621 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" Jul 31 15:09:13.103727 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.103666 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" event={"ID":"ba26204c-c84b-4b1d-94cd-e4be467fdb5b","Type":"ContainerDied","Data":"e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f"} Jul 31 15:09:13.103727 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.103714 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl" event={"ID":"ba26204c-c84b-4b1d-94cd-e4be467fdb5b","Type":"ContainerDied","Data":"b3628d39cf5c7b385e60793dc1c4b4e43be7c4ac7b192770ea688e3c2227deef"} Jul 31 15:09:13.103857 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.103736 2582 scope.go:117] "RemoveContainer" containerID="e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f" Jul 31 15:09:13.113614 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.113597 2582 scope.go:117] "RemoveContainer" containerID="e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f" Jul 31 15:09:13.113855 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:09:13.113838 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f\": container with ID starting with e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f not found: ID does not exist" containerID="e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f" Jul 31 15:09:13.113910 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.113861 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f"} err="failed to get container status \"e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f\": rpc error: code = NotFound desc = could not find container \"e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f\": container with ID starting with e2aa156c91eee882d832e1b56714b878ccad333b8d80678804ba5f2337691e8f not found: ID does not exist" Jul 31 15:09:13.126220 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.126194 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl"] Jul 31 15:09:13.130922 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.130900 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc9xjjl"] Jul 31 15:09:13.872497 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:13.872453 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" podUID="49a081d6-1e2a-4777-b976-4f6a6dfe6016" containerName="main" containerID="cri-o://379dd90d9f250cacd45fb019d331c65906cd038521b9ea3fef8ead736dcc14eb" gracePeriod=45 Jul 31 15:09:14.115308 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.115282 2582 generic.go:356] "Generic (PLEG): container finished" podID="49a081d6-1e2a-4777-b976-4f6a6dfe6016" containerID="379dd90d9f250cacd45fb019d331c65906cd038521b9ea3fef8ead736dcc14eb" exitCode=0 Jul 31 15:09:14.115447 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.115355 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" event={"ID":"49a081d6-1e2a-4777-b976-4f6a6dfe6016","Type":"ContainerDied","Data":"379dd90d9f250cacd45fb019d331c65906cd038521b9ea3fef8ead736dcc14eb"} Jul 31 15:09:14.115447 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.115406 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" event={"ID":"49a081d6-1e2a-4777-b976-4f6a6dfe6016","Type":"ContainerDied","Data":"2130451664ad31876b2487d5aa83cc2c0a99d186adab01d60f1bfccc9c51aac2"} Jul 31 15:09:14.115447 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.115427 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2130451664ad31876b2487d5aa83cc2c0a99d186adab01d60f1bfccc9c51aac2" Jul 31 15:09:14.122407 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.122391 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:09:14.238720 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.238695 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jptn\" (UniqueName: \"kubernetes.io/projected/49a081d6-1e2a-4777-b976-4f6a6dfe6016-kube-api-access-4jptn\") pod \"49a081d6-1e2a-4777-b976-4f6a6dfe6016\" (UID: \"49a081d6-1e2a-4777-b976-4f6a6dfe6016\") " Jul 31 15:09:14.238881 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.238763 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a081d6-1e2a-4777-b976-4f6a6dfe6016-tls-certs\") pod \"49a081d6-1e2a-4777-b976-4f6a6dfe6016\" (UID: \"49a081d6-1e2a-4777-b976-4f6a6dfe6016\") " Jul 31 15:09:14.240813 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.240788 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a081d6-1e2a-4777-b976-4f6a6dfe6016-kube-api-access-4jptn" (OuterVolumeSpecName: "kube-api-access-4jptn") pod "49a081d6-1e2a-4777-b976-4f6a6dfe6016" (UID: "49a081d6-1e2a-4777-b976-4f6a6dfe6016"). InnerVolumeSpecName "kube-api-access-4jptn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:09:14.240938 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.240906 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a081d6-1e2a-4777-b976-4f6a6dfe6016-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49a081d6-1e2a-4777-b976-4f6a6dfe6016" (UID: "49a081d6-1e2a-4777-b976-4f6a6dfe6016"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:09:14.339682 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.339641 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jptn\" (UniqueName: \"kubernetes.io/projected/49a081d6-1e2a-4777-b976-4f6a6dfe6016-kube-api-access-4jptn\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:09:14.339682 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.339677 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a081d6-1e2a-4777-b976-4f6a6dfe6016-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:09:14.775550 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:14.775517 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba26204c-c84b-4b1d-94cd-e4be467fdb5b" path="/var/lib/kubelet/pods/ba26204c-c84b-4b1d-94cd-e4be467fdb5b/volumes" Jul 31 15:09:15.120484 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:15.120400 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx" Jul 31 15:09:15.137868 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:15.137835 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx"] Jul 31 15:09:15.142312 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:15.142288 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cftg4sx"] Jul 31 15:09:16.742051 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:16.742014 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:16.775406 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:16.775373 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49a081d6-1e2a-4777-b976-4f6a6dfe6016" path="/var/lib/kubelet/pods/49a081d6-1e2a-4777-b976-4f6a6dfe6016/volumes" Jul 31 15:09:16.839737 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:16.839701 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:18.037492 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:18.037449 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:18.144345 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:18.144307 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:23.002019 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:23.001982 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:24.238316 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:24.238281 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:25.630152 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.630120 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:25.808894 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.808863 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd"] Jul 31 15:09:25.809277 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.809263 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba26204c-c84b-4b1d-94cd-e4be467fdb5b" containerName="main" Jul 31 15:09:25.809277 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.809278 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba26204c-c84b-4b1d-94cd-e4be467fdb5b" containerName="main" Jul 31 15:09:25.809412 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.809286 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49a081d6-1e2a-4777-b976-4f6a6dfe6016" containerName="main" Jul 31 15:09:25.809412 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.809291 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="49a081d6-1e2a-4777-b976-4f6a6dfe6016" containerName="main" Jul 31 15:09:25.809412 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.809382 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="49a081d6-1e2a-4777-b976-4f6a6dfe6016" containerName="main" Jul 31 15:09:25.809412 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.809395 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba26204c-c84b-4b1d-94cd-e4be467fdb5b" containerName="main" Jul 31 15:09:25.814245 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.814224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:09:25.816845 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.816807 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\"" Jul 31 15:09:25.817636 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.817620 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-w8lr9\"" Jul 31 15:09:25.817919 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.817900 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\"" Jul 31 15:09:25.818031 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.817949 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-8hshf\"" Jul 31 15:09:25.818178 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.818162 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\"" Jul 31 15:09:25.826313 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.826291 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd"] Jul 31 15:09:25.861016 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.860995 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:25.940050 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.939970 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2kzh\" (UniqueName: \"kubernetes.io/projected/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-kube-api-access-t2kzh\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd\" (UID: \"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:09:25.940191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:25.940068 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd\" (UID: \"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:09:26.041162 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:26.041122 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2kzh\" (UniqueName: \"kubernetes.io/projected/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-kube-api-access-t2kzh\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd\" (UID: \"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:09:26.041318 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:26.041192 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd\" (UID: \"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:09:26.044114 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:26.044090 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd\" (UID: \"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:09:26.049635 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:26.049615 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2kzh\" (UniqueName: \"kubernetes.io/projected/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-kube-api-access-t2kzh\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd\" (UID: \"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:09:26.127062 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:26.127036 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:09:26.260710 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:26.260687 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd"] Jul 31 15:09:26.262607 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:09:26.262576 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ec4e96_12d2_4bc7_b8b2_7a891f262bc9.slice/crio-15c1fca092a6e6c6041d2c955305c6beb1ba2fc638540b31b1151b2bf31d9262 WatchSource:0}: Error finding container 15c1fca092a6e6c6041d2c955305c6beb1ba2fc638540b31b1151b2bf31d9262: Status 404 returned error can't find the container with id 15c1fca092a6e6c6041d2c955305c6beb1ba2fc638540b31b1151b2bf31d9262 Jul 31 15:09:27.191046 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:27.191017 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" event={"ID":"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9","Type":"ContainerStarted","Data":"f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871"} Jul 31 15:09:27.191477 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:27.191066 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" event={"ID":"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9","Type":"ContainerStarted","Data":"15c1fca092a6e6c6041d2c955305c6beb1ba2fc638540b31b1151b2bf31d9262"} Jul 31 15:09:27.191477 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:27.191114 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:09:27.209933 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:27.209888 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" podStartSLOduration=2.209872405 podStartE2EDuration="2.209872405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:09:27.208502917 +0000 UTC m=+1278.809939255" watchObservedRunningTime="2026-07-31 15:09:27.209872405 +0000 UTC m=+1278.811308742" Jul 31 15:09:29.017587 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:29.017544 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:29.118841 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:29.118805 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:30.082330 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:30.082294 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:30.143501 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:30.143475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:31.374254 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:31.374221 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:41.130386 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:41.130349 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:41.229568 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:41.229513 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:58.198086 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:09:58.198060 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:10:00.670576 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:00.670520 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:07.004069 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:07.004030 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:14.225011 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:14.224975 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:14.326302 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:14.326277 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:34.254375 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:34.254343 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:34.354258 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:34.354230 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:38.399046 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:38.399013 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:38.624530 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:38.624501 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:42.408602 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:42.408569 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:43.954646 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:43.954613 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:44.007764 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:44.007735 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:44.323486 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:44.323403 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:46.915764 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:46.915735 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:48.315978 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:48.315949 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:48.419061 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:10:48.419029 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:12.476282 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:12.476249 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:12.570440 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:12.570401 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:30.831351 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:30.831315 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd"] Jul 31 15:11:30.831910 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:30.831493 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:30.937710 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:30.937673 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:32.142730 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:32.142696 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:32.239611 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:32.239573 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:38.197968 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:38.197930 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" podUID="92ec4e96-12d2-4bc7-b8b2-7a891f262bc9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:11:38.688383 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:38.688351 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:44.519475 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:44.519439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:44.579449 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:44.579419 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:45.852263 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:45.852229 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" podUID="92ec4e96-12d2-4bc7-b8b2-7a891f262bc9" containerName="main" containerID="cri-o://f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871" gracePeriod=45 Jul 31 15:11:46.099584 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.099547 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:11:46.297235 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.297207 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2kzh\" (UniqueName: \"kubernetes.io/projected/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-kube-api-access-t2kzh\") pod \"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9\" (UID: \"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9\") " Jul 31 15:11:46.297394 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.297254 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-tls-certs\") pod \"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9\" (UID: \"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9\") " Jul 31 15:11:46.299397 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.299369 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92ec4e96-12d2-4bc7-b8b2-7a891f262bc9" (UID: "92ec4e96-12d2-4bc7-b8b2-7a891f262bc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:11:46.299503 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.299416 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-kube-api-access-t2kzh" (OuterVolumeSpecName: "kube-api-access-t2kzh") pod "92ec4e96-12d2-4bc7-b8b2-7a891f262bc9" (UID: "92ec4e96-12d2-4bc7-b8b2-7a891f262bc9"). InnerVolumeSpecName "kube-api-access-t2kzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:11:46.397999 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.397962 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2kzh\" (UniqueName: \"kubernetes.io/projected/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-kube-api-access-t2kzh\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:11:46.397999 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.397995 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:11:46.938572 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.938526 2582 generic.go:356] "Generic (PLEG): container finished" podID="92ec4e96-12d2-4bc7-b8b2-7a891f262bc9" containerID="f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871" exitCode=0 Jul 31 15:11:46.939053 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.938581 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" Jul 31 15:11:46.939053 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.938596 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" event={"ID":"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9","Type":"ContainerDied","Data":"f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871"} Jul 31 15:11:46.939053 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.938654 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd" event={"ID":"92ec4e96-12d2-4bc7-b8b2-7a891f262bc9","Type":"ContainerDied","Data":"15c1fca092a6e6c6041d2c955305c6beb1ba2fc638540b31b1151b2bf31d9262"} Jul 31 15:11:46.939053 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.938679 2582 scope.go:117] "RemoveContainer" containerID="f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871" Jul 31 15:11:46.947581 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.947569 2582 scope.go:117] "RemoveContainer" containerID="f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871" Jul 31 15:11:46.947805 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:11:46.947788 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871\": container with ID starting with f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871 not found: ID does not exist" containerID="f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871" Jul 31 15:11:46.947865 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.947813 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871"} err="failed to get container status \"f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871\": rpc error: code = NotFound desc = could not find container \"f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871\": container with ID starting with f091cb9bcec7da13ce09680697702d192a49e81e92ddbf55d9c6dfca1a055871 not found: ID does not exist" Jul 31 15:11:46.955704 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.955682 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd"] Jul 31 15:11:46.959043 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:46.959024 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-wnnxd"] Jul 31 15:11:48.509434 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.509399 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:48.682315 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.682283 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h"] Jul 31 15:11:48.682864 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.682832 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ec4e96-12d2-4bc7-b8b2-7a891f262bc9" containerName="main" Jul 31 15:11:48.683004 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.682942 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ec4e96-12d2-4bc7-b8b2-7a891f262bc9" containerName="main" Jul 31 15:11:48.683111 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.683096 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ec4e96-12d2-4bc7-b8b2-7a891f262bc9" containerName="main" Jul 31 15:11:48.687564 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.687538 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:11:48.691307 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.691278 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\"" Jul 31 15:11:48.691482 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.691283 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-q2rgg\"" Jul 31 15:11:48.691482 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.691283 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\"" Jul 31 15:11:48.691670 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.691335 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-n9hbs\"" Jul 31 15:11:48.691670 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.691361 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\"" Jul 31 15:11:48.698941 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.698916 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h"] Jul 31 15:11:48.713011 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.712980 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqkw\" (UniqueName: \"kubernetes.io/projected/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-kube-api-access-2cqkw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h\" (UID: \"f16925d0-a09c-43a1-bd90-9aff8d4dddb3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:11:48.713175 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.713112 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h\" (UID: \"f16925d0-a09c-43a1-bd90-9aff8d4dddb3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:11:48.721545 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.721504 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:48.775612 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.775528 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92ec4e96-12d2-4bc7-b8b2-7a891f262bc9" path="/var/lib/kubelet/pods/92ec4e96-12d2-4bc7-b8b2-7a891f262bc9/volumes" Jul 31 15:11:48.813814 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.813783 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h\" (UID: \"f16925d0-a09c-43a1-bd90-9aff8d4dddb3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:11:48.813969 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.813850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqkw\" (UniqueName: \"kubernetes.io/projected/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-kube-api-access-2cqkw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h\" (UID: \"f16925d0-a09c-43a1-bd90-9aff8d4dddb3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:11:48.816207 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.816184 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h\" (UID: \"f16925d0-a09c-43a1-bd90-9aff8d4dddb3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:11:48.822494 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:48.822466 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqkw\" (UniqueName: \"kubernetes.io/projected/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-kube-api-access-2cqkw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h\" (UID: \"f16925d0-a09c-43a1-bd90-9aff8d4dddb3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:11:49.001259 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:49.001222 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:11:49.143386 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:49.143354 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h"] Jul 31 15:11:49.144992 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:11:49.144963 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16925d0_a09c_43a1_bd90_9aff8d4dddb3.slice/crio-9c154d2736d173286714d10dfd94e61978175135f655aeccf3210fe0dcf68b23 WatchSource:0}: Error finding container 9c154d2736d173286714d10dfd94e61978175135f655aeccf3210fe0dcf68b23: Status 404 returned error can't find the container with id 9c154d2736d173286714d10dfd94e61978175135f655aeccf3210fe0dcf68b23 Jul 31 15:11:49.956047 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:49.956012 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" event={"ID":"f16925d0-a09c-43a1-bd90-9aff8d4dddb3","Type":"ContainerStarted","Data":"01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd"} Jul 31 15:11:49.956425 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:49.956154 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:11:49.956425 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:49.956178 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" event={"ID":"f16925d0-a09c-43a1-bd90-9aff8d4dddb3","Type":"ContainerStarted","Data":"9c154d2736d173286714d10dfd94e61978175135f655aeccf3210fe0dcf68b23"} Jul 31 15:11:49.975819 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:49.975772 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" podStartSLOduration=1.9757581819999999 podStartE2EDuration="1.975758182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:11:49.973316475 +0000 UTC m=+1421.574752812" watchObservedRunningTime="2026-07-31 15:11:49.975758182 +0000 UTC m=+1421.577194518" Jul 31 15:11:50.861608 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:50.861577 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:50.963796 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:50.963714 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:51.065027 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:51.064995 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:53.822793 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:53.822762 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:57.417640 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:11:57.417610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:12:03.335886 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:12:03.335816 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:12:03.410171 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:12:03.410144 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:12:13.342637 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:12:13.342600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:12:13.837789 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:12:13.837755 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:12:20.962909 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:12:20.962881 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:12:22.282728 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:12:22.282683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:12:33.322091 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:12:33.322056 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:12:56.725825 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:12:56.725795 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:12:56.743533 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:12:56.743498 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h"] Jul 31 15:12:56.795966 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:12:56.795935 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:00.963579 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:00.963527 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" podUID="f16925d0-a09c-43a1-bd90-9aff8d4dddb3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:13:10.963367 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:10.963330 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" podUID="f16925d0-a09c-43a1-bd90-9aff8d4dddb3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:13:11.771221 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:11.771160 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" podUID="f16925d0-a09c-43a1-bd90-9aff8d4dddb3" containerName="main" containerID="cri-o://01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd" gracePeriod=45 Jul 31 15:13:12.015658 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.015635 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:13:12.076874 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.076817 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cqkw\" (UniqueName: \"kubernetes.io/projected/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-kube-api-access-2cqkw\") pod \"f16925d0-a09c-43a1-bd90-9aff8d4dddb3\" (UID: \"f16925d0-a09c-43a1-bd90-9aff8d4dddb3\") " Jul 31 15:13:12.076874 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.076863 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-tls-certs\") pod \"f16925d0-a09c-43a1-bd90-9aff8d4dddb3\" (UID: \"f16925d0-a09c-43a1-bd90-9aff8d4dddb3\") " Jul 31 15:13:12.078913 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.078887 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-kube-api-access-2cqkw" (OuterVolumeSpecName: "kube-api-access-2cqkw") pod "f16925d0-a09c-43a1-bd90-9aff8d4dddb3" (UID: "f16925d0-a09c-43a1-bd90-9aff8d4dddb3"). InnerVolumeSpecName "kube-api-access-2cqkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:13:12.079020 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.078958 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f16925d0-a09c-43a1-bd90-9aff8d4dddb3" (UID: "f16925d0-a09c-43a1-bd90-9aff8d4dddb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:13:12.177973 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.177946 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqkw\" (UniqueName: \"kubernetes.io/projected/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-kube-api-access-2cqkw\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:13:12.177973 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.177969 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16925d0-a09c-43a1-bd90-9aff8d4dddb3-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:13:12.404415 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.404385 2582 generic.go:356] "Generic (PLEG): container finished" podID="f16925d0-a09c-43a1-bd90-9aff8d4dddb3" containerID="01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd" exitCode=0 Jul 31 15:13:12.404591 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.404430 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" Jul 31 15:13:12.404591 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.404478 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" event={"ID":"f16925d0-a09c-43a1-bd90-9aff8d4dddb3","Type":"ContainerDied","Data":"01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd"} Jul 31 15:13:12.404591 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.404533 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h" event={"ID":"f16925d0-a09c-43a1-bd90-9aff8d4dddb3","Type":"ContainerDied","Data":"9c154d2736d173286714d10dfd94e61978175135f655aeccf3210fe0dcf68b23"} Jul 31 15:13:12.404591 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.404584 2582 scope.go:117] "RemoveContainer" containerID="01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd" Jul 31 15:13:12.414463 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.414444 2582 scope.go:117] "RemoveContainer" containerID="01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd" Jul 31 15:13:12.414744 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:13:12.414723 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd\": container with ID starting with 01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd not found: ID does not exist" containerID="01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd" Jul 31 15:13:12.414798 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.414750 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd"} err="failed to get container status \"01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd\": rpc error: code = NotFound desc = could not find container \"01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd\": container with ID starting with 01f610e46d303d6e907ebae8011092a3d0ce01b8a20ba417ecf921219f5158bd not found: ID does not exist" Jul 31 15:13:12.427608 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.427586 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h"] Jul 31 15:13:12.432275 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.432254 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb7q8h"] Jul 31 15:13:12.776086 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:12.776010 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f16925d0-a09c-43a1-bd90-9aff8d4dddb3" path="/var/lib/kubelet/pods/f16925d0-a09c-43a1-bd90-9aff8d4dddb3/volumes" Jul 31 15:13:17.466165 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:17.466134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:17.567182 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:17.567151 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:23.514540 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:23.514498 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:24.016703 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.016663 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:24.292448 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.292374 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c"] Jul 31 15:13:24.292984 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.292958 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16925d0-a09c-43a1-bd90-9aff8d4dddb3" containerName="main" Jul 31 15:13:24.292984 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.292983 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16925d0-a09c-43a1-bd90-9aff8d4dddb3" containerName="main" Jul 31 15:13:24.293159 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.293057 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f16925d0-a09c-43a1-bd90-9aff8d4dddb3" containerName="main" Jul 31 15:13:24.296238 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.296219 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:13:24.297715 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.297678 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:24.301835 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.301810 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\"" Jul 31 15:13:24.301972 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.301950 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\"" Jul 31 15:13:24.303858 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.303418 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-mdc82\"" Jul 31 15:13:24.303858 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.303436 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-h8q2s\"" Jul 31 15:13:24.303858 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.303699 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\"" Jul 31 15:13:24.306644 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.306624 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c"] Jul 31 15:13:24.376863 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.376837 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4pd\" (UniqueName: \"kubernetes.io/projected/9c617874-a6d9-4b42-82b0-3999ab774725-kube-api-access-mp4pd\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c\" (UID: \"9c617874-a6d9-4b42-82b0-3999ab774725\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:13:24.376968 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.376907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c617874-a6d9-4b42-82b0-3999ab774725-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c\" (UID: \"9c617874-a6d9-4b42-82b0-3999ab774725\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:13:24.389265 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.389245 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:24.477490 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.477461 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4pd\" (UniqueName: \"kubernetes.io/projected/9c617874-a6d9-4b42-82b0-3999ab774725-kube-api-access-mp4pd\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c\" (UID: \"9c617874-a6d9-4b42-82b0-3999ab774725\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:13:24.477657 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.477534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c617874-a6d9-4b42-82b0-3999ab774725-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c\" (UID: \"9c617874-a6d9-4b42-82b0-3999ab774725\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:13:24.479934 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.479919 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c617874-a6d9-4b42-82b0-3999ab774725-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c\" (UID: \"9c617874-a6d9-4b42-82b0-3999ab774725\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:13:24.485716 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.485697 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4pd\" (UniqueName: \"kubernetes.io/projected/9c617874-a6d9-4b42-82b0-3999ab774725-kube-api-access-mp4pd\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c\" (UID: \"9c617874-a6d9-4b42-82b0-3999ab774725\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:13:24.609672 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.609608 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:13:24.745164 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:24.743150 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c"] Jul 31 15:13:24.745626 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:13:24.745591 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c617874_a6d9_4b42_82b0_3999ab774725.slice/crio-2e99a171193b652fe7e2c678fd60c2ad33c34d47cdf4a60cae285a0a6a52a049 WatchSource:0}: Error finding container 2e99a171193b652fe7e2c678fd60c2ad33c34d47cdf4a60cae285a0a6a52a049: Status 404 returned error can't find the container with id 2e99a171193b652fe7e2c678fd60c2ad33c34d47cdf4a60cae285a0a6a52a049 Jul 31 15:13:25.477004 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:25.476968 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" event={"ID":"9c617874-a6d9-4b42-82b0-3999ab774725","Type":"ContainerStarted","Data":"19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78"} Jul 31 15:13:25.477163 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:25.477057 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:13:25.477163 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:25.477086 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" event={"ID":"9c617874-a6d9-4b42-82b0-3999ab774725","Type":"ContainerStarted","Data":"2e99a171193b652fe7e2c678fd60c2ad33c34d47cdf4a60cae285a0a6a52a049"} Jul 31 15:13:25.500803 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:25.500755 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" podStartSLOduration=1.500741313 podStartE2EDuration="1.500741313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:13:25.498875393 +0000 UTC m=+1517.100311721" watchObservedRunningTime="2026-07-31 15:13:25.500741313 +0000 UTC m=+1517.102177649" Jul 31 15:13:29.194256 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:29.194220 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:29.257279 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:29.257240 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:36.163981 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:36.163944 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:56.487465 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:13:56.487436 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:14:20.317816 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:20.317779 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:21.798235 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:21.798200 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:26.398274 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:26.398241 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:27.453923 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:27.453893 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:34.323049 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:34.323010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:48.582581 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:48.582532 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:55.750096 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:55.750057 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:55.844622 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:55.844589 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:56.714653 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:56.714615 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:56.747433 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:56.747407 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c"] Jul 31 15:14:56.786068 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:14:56.786043 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:00.763837 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:00.763805 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:02.354657 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:02.354623 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:02.454384 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:02.454356 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:03.493652 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:03.493618 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:06.487553 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:06.487512 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" podUID="9c617874-a6d9-4b42-82b0-3999ab774725" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:15:08.943907 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:08.943877 2582 scope.go:117] "RemoveContainer" containerID="379dd90d9f250cacd45fb019d331c65906cd038521b9ea3fef8ead736dcc14eb" Jul 31 15:15:11.527778 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:11.527746 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:11.768013 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:11.767956 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" podUID="9c617874-a6d9-4b42-82b0-3999ab774725" containerName="main" containerID="cri-o://19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78" gracePeriod=45 Jul 31 15:15:12.014412 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.014388 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:15:12.049192 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.049117 2582 generic.go:356] "Generic (PLEG): container finished" podID="9c617874-a6d9-4b42-82b0-3999ab774725" containerID="19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78" exitCode=0 Jul 31 15:15:12.049192 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.049160 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" Jul 31 15:15:12.049349 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.049200 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" event={"ID":"9c617874-a6d9-4b42-82b0-3999ab774725","Type":"ContainerDied","Data":"19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78"} Jul 31 15:15:12.049349 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.049274 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c" event={"ID":"9c617874-a6d9-4b42-82b0-3999ab774725","Type":"ContainerDied","Data":"2e99a171193b652fe7e2c678fd60c2ad33c34d47cdf4a60cae285a0a6a52a049"} Jul 31 15:15:12.049349 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.049297 2582 scope.go:117] "RemoveContainer" containerID="19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78" Jul 31 15:15:12.059369 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.059348 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c617874-a6d9-4b42-82b0-3999ab774725-tls-certs\") pod \"9c617874-a6d9-4b42-82b0-3999ab774725\" (UID: \"9c617874-a6d9-4b42-82b0-3999ab774725\") " Jul 31 15:15:12.059484 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.059414 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp4pd\" (UniqueName: \"kubernetes.io/projected/9c617874-a6d9-4b42-82b0-3999ab774725-kube-api-access-mp4pd\") pod \"9c617874-a6d9-4b42-82b0-3999ab774725\" (UID: \"9c617874-a6d9-4b42-82b0-3999ab774725\") " Jul 31 15:15:12.060376 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.060354 2582 scope.go:117] "RemoveContainer" containerID="19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78" Jul 31 15:15:12.060683 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:15:12.060655 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78\": container with ID starting with 19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78 not found: ID does not exist" containerID="19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78" Jul 31 15:15:12.060777 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.060692 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78"} err="failed to get container status \"19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78\": rpc error: code = NotFound desc = could not find container \"19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78\": container with ID starting with 19f1568036545f8b9c8294a4ea5d830f060604959bf8eefd0ca2f2514c34ce78 not found: ID does not exist" Jul 31 15:15:12.061601 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.061579 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c617874-a6d9-4b42-82b0-3999ab774725-kube-api-access-mp4pd" (OuterVolumeSpecName: "kube-api-access-mp4pd") pod "9c617874-a6d9-4b42-82b0-3999ab774725" (UID: "9c617874-a6d9-4b42-82b0-3999ab774725"). InnerVolumeSpecName "kube-api-access-mp4pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:15:12.061834 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.061820 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c617874-a6d9-4b42-82b0-3999ab774725-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c617874-a6d9-4b42-82b0-3999ab774725" (UID: "9c617874-a6d9-4b42-82b0-3999ab774725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:15:12.160721 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.160696 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c617874-a6d9-4b42-82b0-3999ab774725-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:15:12.160721 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.160719 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4pd\" (UniqueName: \"kubernetes.io/projected/9c617874-a6d9-4b42-82b0-3999ab774725-kube-api-access-mp4pd\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:15:12.373694 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.373670 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c"] Jul 31 15:15:12.383405 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.383382 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd652jm7c"] Jul 31 15:15:12.775238 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:12.775200 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c617874-a6d9-4b42-82b0-3999ab774725" path="/var/lib/kubelet/pods/9c617874-a6d9-4b42-82b0-3999ab774725/volumes" Jul 31 15:15:15.906247 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:15.906217 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:16.012504 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:16.012477 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:18.710825 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:18.710786 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:22.568347 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:22.568311 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:27.451855 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:27.451822 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:27.552396 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:27.552362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:27.652518 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:27.652487 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:27.717154 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:27.717066 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:28.499872 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:28.499828 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:28.603620 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:28.603588 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:37.229321 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:37.229289 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:37.325755 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:37.325724 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:38.701089 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:38.701058 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:46.157417 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.157387 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:46.443674 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.443594 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f"] Jul 31 15:15:46.444018 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.444005 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c617874-a6d9-4b42-82b0-3999ab774725" containerName="main" Jul 31 15:15:46.444066 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.444020 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c617874-a6d9-4b42-82b0-3999ab774725" containerName="main" Jul 31 15:15:46.444105 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.444096 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c617874-a6d9-4b42-82b0-3999ab774725" containerName="main" Jul 31 15:15:46.447277 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.447255 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:15:46.447489 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.447425 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:46.450639 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.450622 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-m7dvx\"" Jul 31 15:15:46.450751 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.450627 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\"" Jul 31 15:15:46.450751 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.450682 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\"" Jul 31 15:15:46.450751 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.450629 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\"" Jul 31 15:15:46.450751 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.450724 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-h48tb\"" Jul 31 15:15:46.456344 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.456325 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f"] Jul 31 15:15:46.537632 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.537605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jts29\" (UniqueName: \"kubernetes.io/projected/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-kube-api-access-jts29\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f\" (UID: \"162b2cfc-1cd7-436c-86c4-c8f27d992a5f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:15:46.537779 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.537651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f\" (UID: \"162b2cfc-1cd7-436c-86c4-c8f27d992a5f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:15:46.544388 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.544367 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:46.638551 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.638519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f\" (UID: \"162b2cfc-1cd7-436c-86c4-c8f27d992a5f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:15:46.638733 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.638652 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jts29\" (UniqueName: \"kubernetes.io/projected/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-kube-api-access-jts29\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f\" (UID: \"162b2cfc-1cd7-436c-86c4-c8f27d992a5f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:15:46.640959 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.640929 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f\" (UID: \"162b2cfc-1cd7-436c-86c4-c8f27d992a5f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:15:46.647005 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.646984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jts29\" (UniqueName: \"kubernetes.io/projected/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-kube-api-access-jts29\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f\" (UID: \"162b2cfc-1cd7-436c-86c4-c8f27d992a5f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:15:46.759450 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.759397 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:15:46.896271 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:46.896244 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f"] Jul 31 15:15:46.898236 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:15:46.898196 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162b2cfc_1cd7_436c_86c4_c8f27d992a5f.slice/crio-6931ba7d9fcfcb8ac72c5f70f977785e42d499734863e772280f57b43e0b2dab WatchSource:0}: Error finding container 6931ba7d9fcfcb8ac72c5f70f977785e42d499734863e772280f57b43e0b2dab: Status 404 returned error can't find the container with id 6931ba7d9fcfcb8ac72c5f70f977785e42d499734863e772280f57b43e0b2dab Jul 31 15:15:47.245655 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:47.245620 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" event={"ID":"162b2cfc-1cd7-436c-86c4-c8f27d992a5f","Type":"ContainerStarted","Data":"27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98"} Jul 31 15:15:47.246040 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:47.245672 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" event={"ID":"162b2cfc-1cd7-436c-86c4-c8f27d992a5f","Type":"ContainerStarted","Data":"6931ba7d9fcfcb8ac72c5f70f977785e42d499734863e772280f57b43e0b2dab"} Jul 31 15:15:47.246040 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:47.245743 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:15:47.268911 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:15:47.268862 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" podStartSLOduration=1.268850344 podStartE2EDuration="1.268850344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:47.267815847 +0000 UTC m=+1658.869252186" watchObservedRunningTime="2026-07-31 15:15:47.268850344 +0000 UTC m=+1658.870286681" Jul 31 15:16:10.410570 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:10.410523 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:11.982399 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:11.982362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:12.079798 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:12.079757 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:18.253016 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:18.252991 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:16:23.447063 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:23.447028 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:23.464384 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:23.464336 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f"] Jul 31 15:16:23.510829 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:23.510794 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:25.829036 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:25.829004 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:25.936223 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:25.936194 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:26.091117 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:26.091028 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:26.188782 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:26.188742 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:28.253155 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:28.253116 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" podUID="162b2cfc-1cd7-436c-86c4-c8f27d992a5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:16:30.867948 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:30.867878 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:31.854732 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:31.854700 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:33.026043 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:33.026010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:37.098408 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:37.098365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:38.253514 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:38.253476 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" podUID="162b2cfc-1cd7-436c-86c4-c8f27d992a5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:16:38.488001 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:38.487960 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" podUID="162b2cfc-1cd7-436c-86c4-c8f27d992a5f" containerName="main" containerID="cri-o://27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98" gracePeriod=45 Jul 31 15:16:38.743275 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:38.743253 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:16:38.863293 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:38.863205 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-tls-certs\") pod \"162b2cfc-1cd7-436c-86c4-c8f27d992a5f\" (UID: \"162b2cfc-1cd7-436c-86c4-c8f27d992a5f\") " Jul 31 15:16:38.863293 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:38.863247 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jts29\" (UniqueName: \"kubernetes.io/projected/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-kube-api-access-jts29\") pod \"162b2cfc-1cd7-436c-86c4-c8f27d992a5f\" (UID: \"162b2cfc-1cd7-436c-86c4-c8f27d992a5f\") " Jul 31 15:16:38.865457 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:38.865423 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-kube-api-access-jts29" (OuterVolumeSpecName: "kube-api-access-jts29") pod "162b2cfc-1cd7-436c-86c4-c8f27d992a5f" (UID: "162b2cfc-1cd7-436c-86c4-c8f27d992a5f"). InnerVolumeSpecName "kube-api-access-jts29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:16:38.865600 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:38.865459 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "162b2cfc-1cd7-436c-86c4-c8f27d992a5f" (UID: "162b2cfc-1cd7-436c-86c4-c8f27d992a5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:16:38.964327 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:38.964300 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:16:38.964327 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:38.964324 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jts29\" (UniqueName: \"kubernetes.io/projected/162b2cfc-1cd7-436c-86c4-c8f27d992a5f-kube-api-access-jts29\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:16:39.404331 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:39.404287 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:39.549110 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:39.549078 2582 generic.go:356] "Generic (PLEG): container finished" podID="162b2cfc-1cd7-436c-86c4-c8f27d992a5f" containerID="27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98" exitCode=0 Jul 31 15:16:39.549282 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:39.549121 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" Jul 31 15:16:39.549282 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:39.549161 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" event={"ID":"162b2cfc-1cd7-436c-86c4-c8f27d992a5f","Type":"ContainerDied","Data":"27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98"} Jul 31 15:16:39.549282 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:39.549213 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f" event={"ID":"162b2cfc-1cd7-436c-86c4-c8f27d992a5f","Type":"ContainerDied","Data":"6931ba7d9fcfcb8ac72c5f70f977785e42d499734863e772280f57b43e0b2dab"} Jul 31 15:16:39.549282 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:39.549237 2582 scope.go:117] "RemoveContainer" containerID="27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98" Jul 31 15:16:39.558130 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:39.558110 2582 scope.go:117] "RemoveContainer" containerID="27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98" Jul 31 15:16:39.558376 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:16:39.558358 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98\": container with ID starting with 27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98 not found: ID does not exist" containerID="27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98" Jul 31 15:16:39.558437 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:39.558387 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98"} err="failed to get container status \"27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98\": rpc error: code = NotFound desc = could not find container \"27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98\": container with ID starting with 27bab0f9facdf0e1186e18a9fc9d6451a71363aea2e09064b732094e2bdf5e98 not found: ID does not exist" Jul 31 15:16:39.572848 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:39.572826 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f"] Jul 31 15:16:39.576518 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:39.576497 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9bqb4f"] Jul 31 15:16:40.022735 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:40.022703 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:40.775472 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:40.775439 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="162b2cfc-1cd7-436c-86c4-c8f27d992a5f" path="/var/lib/kubelet/pods/162b2cfc-1cd7-436c-86c4-c8f27d992a5f/volumes" Jul 31 15:16:43.275953 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:43.275920 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:43.958622 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:43.958595 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:49.733196 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:49.733162 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:56.254969 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:56.254941 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:56.363970 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:16:56.363942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:07.389456 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:07.389421 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:07.487347 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:07.487315 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:08.641534 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:08.641500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:16.318140 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.318103 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv"] Jul 31 15:17:16.318507 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.318478 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="162b2cfc-1cd7-436c-86c4-c8f27d992a5f" containerName="main" Jul 31 15:17:16.318507 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.318489 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="162b2cfc-1cd7-436c-86c4-c8f27d992a5f" containerName="main" Jul 31 15:17:16.318617 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.318572 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="162b2cfc-1cd7-436c-86c4-c8f27d992a5f" containerName="main" Jul 31 15:17:16.321552 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.321534 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.325205 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.325185 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-pkbzj\"" Jul 31 15:17:16.325922 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.325900 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\"" Jul 31 15:17:16.326039 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.325900 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\"" Jul 31 15:17:16.326039 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.325949 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\"" Jul 31 15:17:16.349363 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.349340 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv"] Jul 31 15:17:16.365379 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.365357 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:16.381928 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.381904 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.382031 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.381972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.382100 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.382032 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230a45af-bd26-43de-8f2b-dad19cb88c9c-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.382100 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.382067 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.382207 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.382111 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/230a45af-bd26-43de-8f2b-dad19cb88c9c-kube-api-access-b5lhj\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.382207 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.382153 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.483073 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.483049 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.483198 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.483095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230a45af-bd26-43de-8f2b-dad19cb88c9c-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.483198 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.483117 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.483198 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.483149 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/230a45af-bd26-43de-8f2b-dad19cb88c9c-kube-api-access-b5lhj\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.483198 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.483168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.483427 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.483214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.483593 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.483542 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.483593 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.483581 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.483728 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.483625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.485390 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.485365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.485641 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.485623 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230a45af-bd26-43de-8f2b-dad19cb88c9c-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.492235 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.492208 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/230a45af-bd26-43de-8f2b-dad19cb88c9c-kube-api-access-b5lhj\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.631885 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.631850 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:16.638485 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.638455 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:16.775950 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:17:16.775913 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230a45af_bd26_43de_8f2b_dad19cb88c9c.slice/crio-98ed9decf0439040779c4daa14220a23537364a339a49156e872b4e4ae2edee4 WatchSource:0}: Error finding container 98ed9decf0439040779c4daa14220a23537364a339a49156e872b4e4ae2edee4: Status 404 returned error can't find the container with id 98ed9decf0439040779c4daa14220a23537364a339a49156e872b4e4ae2edee4 Jul 31 15:17:16.777330 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:16.777308 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv"] Jul 31 15:17:17.755080 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:17.755037 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" event={"ID":"230a45af-bd26-43de-8f2b-dad19cb88c9c","Type":"ContainerStarted","Data":"6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813"} Jul 31 15:17:17.755464 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:17.755134 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" event={"ID":"230a45af-bd26-43de-8f2b-dad19cb88c9c","Type":"ContainerStarted","Data":"98ed9decf0439040779c4daa14220a23537364a339a49156e872b4e4ae2edee4"} Jul 31 15:17:17.774994 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:17.774947 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" podStartSLOduration=1.7749321679999999 podStartE2EDuration="1.774932168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:17:17.77327382 +0000 UTC m=+1749.374710157" watchObservedRunningTime="2026-07-31 15:17:17.774932168 +0000 UTC m=+1749.376368513" Jul 31 15:17:18.632495 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:18.632460 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:24.211921 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:24.211887 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:24.315041 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:24.315006 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:26.632514 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:26.632478 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:26.644583 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:26.644543 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:26.820123 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:26.820095 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:17:28.485843 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:28.485787 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:30.101761 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:30.101725 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:50.112688 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:50.112655 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:50.215430 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:50.215389 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:50.320033 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:50.320001 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:55.280928 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:17:55.280897 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:06.444892 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:06.444864 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:29.099261 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:29.099227 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:32.221510 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:32.221480 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:33.204647 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:33.204613 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:38.050697 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.050664 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:38.053044 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.053014 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv"] Jul 31 15:18:38.062507 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:18:38.062473 2582 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 31 15:18:38.062507 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:18:38.062507 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:18:38.062507 ip-10-0-135-143 kubenswrapper[2582]: > containerID="6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813" cmd=["/bin/sleep","15"] Jul 31 15:18:38.062744 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.062521 2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" message="" err=< Jul 31 15:18:38.062744 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:18:38.062744 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:18:38.062744 ip-10-0-135-143 kubenswrapper[2582]: > Jul 31 15:18:38.062744 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:18:38.062535 2582 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 31 15:18:38.062744 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:18:38.062744 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:18:38.062744 ip-10-0-135-143 kubenswrapper[2582]: > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" podUID="230a45af-bd26-43de-8f2b-dad19cb88c9c" containerName="main" containerID="cri-o://6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813" Jul 31 15:18:38.062744 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.062632 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" podUID="230a45af-bd26-43de-8f2b-dad19cb88c9c" containerName="main" containerID="cri-o://6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813" gracePeriod=60 Jul 31 15:18:38.110765 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.110728 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:38.413545 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.413523 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:18:38.547188 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547161 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/230a45af-bd26-43de-8f2b-dad19cb88c9c-kube-api-access-b5lhj\") pod \"230a45af-bd26-43de-8f2b-dad19cb88c9c\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " Jul 31 15:18:38.547324 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547196 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-model-cache\") pod \"230a45af-bd26-43de-8f2b-dad19cb88c9c\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " Jul 31 15:18:38.547324 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547234 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-dshm\") pod \"230a45af-bd26-43de-8f2b-dad19cb88c9c\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " Jul 31 15:18:38.547401 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547337 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-home\") pod \"230a45af-bd26-43de-8f2b-dad19cb88c9c\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " Jul 31 15:18:38.547401 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547385 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-tmp-dir\") pod \"230a45af-bd26-43de-8f2b-dad19cb88c9c\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " Jul 31 15:18:38.547469 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547412 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "230a45af-bd26-43de-8f2b-dad19cb88c9c" (UID: "230a45af-bd26-43de-8f2b-dad19cb88c9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:18:38.547469 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547425 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230a45af-bd26-43de-8f2b-dad19cb88c9c-tls-certs\") pod \"230a45af-bd26-43de-8f2b-dad19cb88c9c\" (UID: \"230a45af-bd26-43de-8f2b-dad19cb88c9c\") " Jul 31 15:18:38.547596 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547576 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "230a45af-bd26-43de-8f2b-dad19cb88c9c" (UID: "230a45af-bd26-43de-8f2b-dad19cb88c9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:18:38.547645 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547597 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-home" (OuterVolumeSpecName: "home") pod "230a45af-bd26-43de-8f2b-dad19cb88c9c" (UID: "230a45af-bd26-43de-8f2b-dad19cb88c9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:18:38.547756 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547736 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:18:38.547804 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547764 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:18:38.547804 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.547778 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:18:38.549480 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.549456 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230a45af-bd26-43de-8f2b-dad19cb88c9c-kube-api-access-b5lhj" (OuterVolumeSpecName: "kube-api-access-b5lhj") pod "230a45af-bd26-43de-8f2b-dad19cb88c9c" (UID: "230a45af-bd26-43de-8f2b-dad19cb88c9c"). InnerVolumeSpecName "kube-api-access-b5lhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:18:38.549584 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.549458 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-dshm" (OuterVolumeSpecName: "dshm") pod "230a45af-bd26-43de-8f2b-dad19cb88c9c" (UID: "230a45af-bd26-43de-8f2b-dad19cb88c9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:18:38.549643 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.549585 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230a45af-bd26-43de-8f2b-dad19cb88c9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "230a45af-bd26-43de-8f2b-dad19cb88c9c" (UID: "230a45af-bd26-43de-8f2b-dad19cb88c9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:18:38.648897 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.648865 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230a45af-bd26-43de-8f2b-dad19cb88c9c-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:18:38.648897 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.648895 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/230a45af-bd26-43de-8f2b-dad19cb88c9c-kube-api-access-b5lhj\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:18:38.648897 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:38.648905 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230a45af-bd26-43de-8f2b-dad19cb88c9c-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:18:39.200035 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:39.200003 2582 generic.go:356] "Generic (PLEG): container finished" podID="230a45af-bd26-43de-8f2b-dad19cb88c9c" containerID="6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813" exitCode=0 Jul 31 15:18:39.200536 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:39.200053 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" event={"ID":"230a45af-bd26-43de-8f2b-dad19cb88c9c","Type":"ContainerDied","Data":"6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813"} Jul 31 15:18:39.200536 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:39.200062 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" Jul 31 15:18:39.200536 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:39.200095 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv" event={"ID":"230a45af-bd26-43de-8f2b-dad19cb88c9c","Type":"ContainerDied","Data":"98ed9decf0439040779c4daa14220a23537364a339a49156e872b4e4ae2edee4"} Jul 31 15:18:39.200536 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:39.200116 2582 scope.go:117] "RemoveContainer" containerID="6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813" Jul 31 15:18:39.211397 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:39.211380 2582 scope.go:117] "RemoveContainer" containerID="6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813" Jul 31 15:18:39.211717 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:18:39.211690 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813\": container with ID starting with 6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813 not found: ID does not exist" containerID="6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813" Jul 31 15:18:39.211776 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:39.211725 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813"} err="failed to get container status \"6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813\": rpc error: code = NotFound desc = could not find container \"6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813\": container with ID starting with 6005c90bf74f82abbbac8aa645b2d8315610017a2197f9bc80a7bcaf3f973813 not found: ID does not exist" Jul 31 15:18:39.223756 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:39.223735 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv"] Jul 31 15:18:39.227720 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:39.227698 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fqzv"] Jul 31 15:18:40.775305 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:40.775274 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230a45af-bd26-43de-8f2b-dad19cb88c9c" path="/var/lib/kubelet/pods/230a45af-bd26-43de-8f2b-dad19cb88c9c/volumes" Jul 31 15:18:57.029168 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:57.029138 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:57.131883 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:57.131845 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:57.229097 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:57.229068 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:59.336350 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.336319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:59.563259 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.563223 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487"] Jul 31 15:18:59.563868 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.563837 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230a45af-bd26-43de-8f2b-dad19cb88c9c" containerName="main" Jul 31 15:18:59.563868 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.563864 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="230a45af-bd26-43de-8f2b-dad19cb88c9c" containerName="main" Jul 31 15:18:59.563996 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.563984 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="230a45af-bd26-43de-8f2b-dad19cb88c9c" containerName="main" Jul 31 15:18:59.567041 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.567020 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:18:59.573312 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.572730 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\"" Jul 31 15:18:59.573312 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.573091 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\"" Jul 31 15:18:59.573494 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.573429 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-4r8l8\"" Jul 31 15:18:59.574572 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.573684 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\"" Jul 31 15:18:59.574572 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.573928 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-qptkq\"" Jul 31 15:18:59.576121 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.576036 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487"] Jul 31 15:18:59.604974 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.604948 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a583a9fb-409d-417f-b0f9-0e3def7d7986-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487\" (UID: \"a583a9fb-409d-417f-b0f9-0e3def7d7986\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:18:59.605085 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.605003 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfn2\" (UniqueName: \"kubernetes.io/projected/a583a9fb-409d-417f-b0f9-0e3def7d7986-kube-api-access-kqfn2\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487\" (UID: \"a583a9fb-409d-417f-b0f9-0e3def7d7986\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:18:59.605749 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.605723 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:59.705664 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.705641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a583a9fb-409d-417f-b0f9-0e3def7d7986-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487\" (UID: \"a583a9fb-409d-417f-b0f9-0e3def7d7986\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:18:59.705756 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.705688 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfn2\" (UniqueName: \"kubernetes.io/projected/a583a9fb-409d-417f-b0f9-0e3def7d7986-kube-api-access-kqfn2\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487\" (UID: \"a583a9fb-409d-417f-b0f9-0e3def7d7986\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:18:59.708115 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.708087 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a583a9fb-409d-417f-b0f9-0e3def7d7986-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487\" (UID: \"a583a9fb-409d-417f-b0f9-0e3def7d7986\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:18:59.715023 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.715002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfn2\" (UniqueName: \"kubernetes.io/projected/a583a9fb-409d-417f-b0f9-0e3def7d7986-kube-api-access-kqfn2\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487\" (UID: \"a583a9fb-409d-417f-b0f9-0e3def7d7986\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:18:59.888450 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:18:59.888423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:19:00.019994 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:00.019969 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487"] Jul 31 15:19:00.021807 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:19:00.021771 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda583a9fb_409d_417f_b0f9_0e3def7d7986.slice/crio-c0841db649e3d26e7a4e0343320f94c1e98d570e1a3246672fe1a786925a0660 WatchSource:0}: Error finding container c0841db649e3d26e7a4e0343320f94c1e98d570e1a3246672fe1a786925a0660: Status 404 returned error can't find the container with id c0841db649e3d26e7a4e0343320f94c1e98d570e1a3246672fe1a786925a0660 Jul 31 15:19:00.317676 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:00.317589 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" event={"ID":"a583a9fb-409d-417f-b0f9-0e3def7d7986","Type":"ContainerStarted","Data":"394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a"} Jul 31 15:19:00.317676 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:00.317652 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" event={"ID":"a583a9fb-409d-417f-b0f9-0e3def7d7986","Type":"ContainerStarted","Data":"c0841db649e3d26e7a4e0343320f94c1e98d570e1a3246672fe1a786925a0660"} Jul 31 15:19:00.317930 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:00.317684 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:19:00.335714 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:00.335671 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" podStartSLOduration=1.335657116 podStartE2EDuration="1.335657116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:19:00.334865379 +0000 UTC m=+1851.936301717" watchObservedRunningTime="2026-07-31 15:19:00.335657116 +0000 UTC m=+1851.937093452" Jul 31 15:19:04.418122 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:04.418089 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:10.447443 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:10.447410 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:10.505111 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:10.505088 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:10.761163 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:10.761088 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:13.049244 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:13.049214 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:15.547469 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:15.547438 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:31.324519 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:31.324439 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:19:31.533653 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:31.533622 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:33.044025 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:33.043992 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:36.610933 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:36.610888 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:36.917665 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:36.917635 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:41.290994 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:41.290962 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:41.393182 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:41.393150 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:44.626631 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:44.626592 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:59.335416 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:59.335382 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:19:59.592980 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:19:59.592938 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:20:12.762388 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:20:12.762361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:20:14.084801 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:20:14.084759 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:20:19.898075 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:20:19.898047 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:20:30.088041 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:20:30.088004 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:20:30.189136 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:20:30.189103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:20:34.544372 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:20:34.544333 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:20:53.344627 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:20:53.344589 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:20:58.025606 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:20:58.025574 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:08.924267 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:08.924191 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:12.063620 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:12.063547 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:12.121493 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:12.121460 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:12.220268 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:12.220241 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:23.917114 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:23.917083 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:23.973689 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:23.973659 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:27.519919 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:27.519888 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:30.510114 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:30.510077 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:33.599683 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:33.599653 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:39.798394 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:39.798357 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:39.902170 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:39.902144 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:42.817363 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:42.817333 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:42.906787 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:42.906756 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487"] Jul 31 15:21:42.924713 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:42.924685 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:51.324597 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:51.324544 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" podUID="a583a9fb-409d-417f-b0f9-0e3def7d7986" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:21:57.449346 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:57.449310 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:21:57.925966 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:57.925927 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" podUID="a583a9fb-409d-417f-b0f9-0e3def7d7986" containerName="main" containerID="cri-o://394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a" gracePeriod=45 Jul 31 15:21:58.174053 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.174030 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:21:58.199608 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.199506 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqfn2\" (UniqueName: \"kubernetes.io/projected/a583a9fb-409d-417f-b0f9-0e3def7d7986-kube-api-access-kqfn2\") pod \"a583a9fb-409d-417f-b0f9-0e3def7d7986\" (UID: \"a583a9fb-409d-417f-b0f9-0e3def7d7986\") " Jul 31 15:21:58.199608 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.199596 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a583a9fb-409d-417f-b0f9-0e3def7d7986-tls-certs\") pod \"a583a9fb-409d-417f-b0f9-0e3def7d7986\" (UID: \"a583a9fb-409d-417f-b0f9-0e3def7d7986\") " Jul 31 15:21:58.201614 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.201544 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a583a9fb-409d-417f-b0f9-0e3def7d7986-kube-api-access-kqfn2" (OuterVolumeSpecName: "kube-api-access-kqfn2") pod "a583a9fb-409d-417f-b0f9-0e3def7d7986" (UID: "a583a9fb-409d-417f-b0f9-0e3def7d7986"). InnerVolumeSpecName "kube-api-access-kqfn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:21:58.201614 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.201591 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a583a9fb-409d-417f-b0f9-0e3def7d7986-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a583a9fb-409d-417f-b0f9-0e3def7d7986" (UID: "a583a9fb-409d-417f-b0f9-0e3def7d7986"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:21:58.274046 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.274020 2582 generic.go:356] "Generic (PLEG): container finished" podID="a583a9fb-409d-417f-b0f9-0e3def7d7986" containerID="394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a" exitCode=0 Jul 31 15:21:58.274193 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.274072 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" Jul 31 15:21:58.274193 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.274103 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" event={"ID":"a583a9fb-409d-417f-b0f9-0e3def7d7986","Type":"ContainerDied","Data":"394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a"} Jul 31 15:21:58.274193 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.274154 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487" event={"ID":"a583a9fb-409d-417f-b0f9-0e3def7d7986","Type":"ContainerDied","Data":"c0841db649e3d26e7a4e0343320f94c1e98d570e1a3246672fe1a786925a0660"} Jul 31 15:21:58.274193 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.274175 2582 scope.go:117] "RemoveContainer" containerID="394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a" Jul 31 15:21:58.283651 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.283633 2582 scope.go:117] "RemoveContainer" containerID="394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a" Jul 31 15:21:58.283906 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:21:58.283889 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a\": container with ID starting with 394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a not found: ID does not exist" containerID="394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a" Jul 31 15:21:58.283979 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.283914 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a"} err="failed to get container status \"394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a\": rpc error: code = NotFound desc = could not find container \"394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a\": container with ID starting with 394da817ae25d80df44ccaaf28751c3cbfc2bde582ecdb0da6a09e78f1cb878a not found: ID does not exist" Jul 31 15:21:58.296992 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.296971 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487"] Jul 31 15:21:58.300328 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.300302 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqfn2\" (UniqueName: \"kubernetes.io/projected/a583a9fb-409d-417f-b0f9-0e3def7d7986-kube-api-access-kqfn2\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:21:58.300328 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.300327 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a583a9fb-409d-417f-b0f9-0e3def7d7986-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:21:58.303090 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.303069 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevm487"] Jul 31 15:21:58.775863 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:21:58.775832 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a583a9fb-409d-417f-b0f9-0e3def7d7986" path="/var/lib/kubelet/pods/a583a9fb-409d-417f-b0f9-0e3def7d7986/volumes" Jul 31 15:22:02.728366 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:02.728306 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:02.813888 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:02.813858 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:06.737762 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.737730 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:06.975401 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.975366 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn"] Jul 31 15:22:06.975766 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.975753 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a583a9fb-409d-417f-b0f9-0e3def7d7986" containerName="main" Jul 31 15:22:06.975825 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.975767 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a583a9fb-409d-417f-b0f9-0e3def7d7986" containerName="main" Jul 31 15:22:06.975868 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.975835 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a583a9fb-409d-417f-b0f9-0e3def7d7986" containerName="main" Jul 31 15:22:06.980259 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.980236 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:22:06.983649 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.983622 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\"" Jul 31 15:22:06.983798 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.983662 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\"" Jul 31 15:22:06.983798 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.983627 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\"" Jul 31 15:22:06.983798 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.983624 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-4dwwj\"" Jul 31 15:22:06.983798 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.983627 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-zrs6n\"" Jul 31 15:22:06.989104 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:06.989047 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn"] Jul 31 15:22:07.004341 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:07.004318 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:07.080761 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:07.080735 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de18341-e1b9-444e-85e8-b088d067d4c9-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn\" (UID: \"3de18341-e1b9-444e-85e8-b088d067d4c9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:22:07.080918 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:07.080770 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkhrk\" (UniqueName: \"kubernetes.io/projected/3de18341-e1b9-444e-85e8-b088d067d4c9-kube-api-access-qkhrk\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn\" (UID: \"3de18341-e1b9-444e-85e8-b088d067d4c9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:22:07.181783 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:07.181751 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de18341-e1b9-444e-85e8-b088d067d4c9-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn\" (UID: \"3de18341-e1b9-444e-85e8-b088d067d4c9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:22:07.181930 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:07.181789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkhrk\" (UniqueName: \"kubernetes.io/projected/3de18341-e1b9-444e-85e8-b088d067d4c9-kube-api-access-qkhrk\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn\" (UID: \"3de18341-e1b9-444e-85e8-b088d067d4c9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:22:07.184371 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:07.184346 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de18341-e1b9-444e-85e8-b088d067d4c9-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn\" (UID: \"3de18341-e1b9-444e-85e8-b088d067d4c9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:22:07.190520 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:07.190494 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkhrk\" (UniqueName: \"kubernetes.io/projected/3de18341-e1b9-444e-85e8-b088d067d4c9-kube-api-access-qkhrk\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn\" (UID: \"3de18341-e1b9-444e-85e8-b088d067d4c9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:22:07.291725 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:07.291656 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:22:07.428197 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:07.428171 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn"] Jul 31 15:22:07.429840 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:22:07.429807 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de18341_e1b9_444e_85e8_b088d067d4c9.slice/crio-491baa7c33f47bd78ce745c9c26741423e7ea0877bf4e3ea2471337ae8422b5c WatchSource:0}: Error finding container 491baa7c33f47bd78ce745c9c26741423e7ea0877bf4e3ea2471337ae8422b5c: Status 404 returned error can't find the container with id 491baa7c33f47bd78ce745c9c26741423e7ea0877bf4e3ea2471337ae8422b5c Jul 31 15:22:08.330673 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:08.330634 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" event={"ID":"3de18341-e1b9-444e-85e8-b088d067d4c9","Type":"ContainerStarted","Data":"4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2"} Jul 31 15:22:08.331071 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:08.330686 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" event={"ID":"3de18341-e1b9-444e-85e8-b088d067d4c9","Type":"ContainerStarted","Data":"491baa7c33f47bd78ce745c9c26741423e7ea0877bf4e3ea2471337ae8422b5c"} Jul 31 15:22:08.331071 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:08.330738 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:22:08.349242 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:08.349182 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" podStartSLOduration=2.349168535 podStartE2EDuration="2.349168535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:22:08.348103607 +0000 UTC m=+2039.949539959" watchObservedRunningTime="2026-07-31 15:22:08.349168535 +0000 UTC m=+2039.950604872" Jul 31 15:22:10.176075 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:10.176044 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:14.466854 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:14.466808 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:14.572907 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:14.572874 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:14.674912 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:14.674882 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:16.265747 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:16.265716 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:16.372781 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:16.372743 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:19.540380 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:19.540347 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:21.952850 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:21.952820 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:35.042029 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:35.041998 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:39.336895 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:39.336865 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:22:56.661527 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:56.661497 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:56.761585 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:56.761540 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:22:56.957524 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:22:56.957444 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:23:09.204523 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:23:09.204489 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:23:09.468134 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:23:09.468051 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:23:20.977540 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:23:20.977508 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:23:21.059699 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:23:21.059665 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:23:33.656722 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:23:33.656692 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:23:47.923378 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:23:47.923342 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:23:59.818436 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:23:59.818402 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:23:59.915141 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:23:59.915092 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:15.570850 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:15.570749 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:15.587209 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:15.587175 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn"] Jul 31 15:24:15.632676 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:15.632648 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:19.337458 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:19.337422 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" podUID="3de18341-e1b9-444e-85e8-b088d067d4c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:24:27.871149 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:27.871112 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:27.970423 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:27.970363 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:28.073734 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:28.073697 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:29.337056 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:29.337015 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" podUID="3de18341-e1b9-444e-85e8-b088d067d4c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:24:30.610031 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:30.609987 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" podUID="3de18341-e1b9-444e-85e8-b088d067d4c9" containerName="main" containerID="cri-o://4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2" gracePeriod=45 Jul 31 15:24:30.866420 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:30.866364 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:24:30.956587 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:30.956531 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkhrk\" (UniqueName: \"kubernetes.io/projected/3de18341-e1b9-444e-85e8-b088d067d4c9-kube-api-access-qkhrk\") pod \"3de18341-e1b9-444e-85e8-b088d067d4c9\" (UID: \"3de18341-e1b9-444e-85e8-b088d067d4c9\") " Jul 31 15:24:30.956751 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:30.956665 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de18341-e1b9-444e-85e8-b088d067d4c9-tls-certs\") pod \"3de18341-e1b9-444e-85e8-b088d067d4c9\" (UID: \"3de18341-e1b9-444e-85e8-b088d067d4c9\") " Jul 31 15:24:30.958749 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:30.958717 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de18341-e1b9-444e-85e8-b088d067d4c9-kube-api-access-qkhrk" (OuterVolumeSpecName: "kube-api-access-qkhrk") pod "3de18341-e1b9-444e-85e8-b088d067d4c9" (UID: "3de18341-e1b9-444e-85e8-b088d067d4c9"). InnerVolumeSpecName "kube-api-access-qkhrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:24:30.958880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:30.958724 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de18341-e1b9-444e-85e8-b088d067d4c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3de18341-e1b9-444e-85e8-b088d067d4c9" (UID: "3de18341-e1b9-444e-85e8-b088d067d4c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:24:31.058161 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.058131 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de18341-e1b9-444e-85e8-b088d067d4c9-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:24:31.058161 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.058158 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkhrk\" (UniqueName: \"kubernetes.io/projected/3de18341-e1b9-444e-85e8-b088d067d4c9-kube-api-access-qkhrk\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:24:31.089199 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.089164 2582 generic.go:356] "Generic (PLEG): container finished" podID="3de18341-e1b9-444e-85e8-b088d067d4c9" containerID="4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2" exitCode=0 Jul 31 15:24:31.089395 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.089210 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" Jul 31 15:24:31.089395 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.089228 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" event={"ID":"3de18341-e1b9-444e-85e8-b088d067d4c9","Type":"ContainerDied","Data":"4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2"} Jul 31 15:24:31.089395 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.089276 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn" event={"ID":"3de18341-e1b9-444e-85e8-b088d067d4c9","Type":"ContainerDied","Data":"491baa7c33f47bd78ce745c9c26741423e7ea0877bf4e3ea2471337ae8422b5c"} Jul 31 15:24:31.089395 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.089301 2582 scope.go:117] "RemoveContainer" containerID="4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2" Jul 31 15:24:31.098817 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.098799 2582 scope.go:117] "RemoveContainer" containerID="4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2" Jul 31 15:24:31.099078 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:24:31.099058 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2\": container with ID starting with 4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2 not found: ID does not exist" containerID="4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2" Jul 31 15:24:31.099140 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.099087 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2"} err="failed to get container status \"4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2\": rpc error: code = NotFound desc = could not find container \"4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2\": container with ID starting with 4a88aba166656b32c2dca4e73ed8916751a1c772dd3d7fe28dedb0a88e7a60b2 not found: ID does not exist" Jul 31 15:24:31.112268 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.112248 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn"] Jul 31 15:24:31.116564 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:31.116509 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5v4xn"] Jul 31 15:24:32.776575 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:32.776538 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de18341-e1b9-444e-85e8-b088d067d4c9" path="/var/lib/kubelet/pods/3de18341-e1b9-444e-85e8-b088d067d4c9/volumes" Jul 31 15:24:32.925894 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:32.925858 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:35.528380 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:35.528348 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:36.837794 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:36.837761 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:39.441270 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:39.441234 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:43.338598 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:43.338549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:49.220539 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:49.220506 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:49.367223 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:49.367193 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:24:49.467231 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:24:49.467200 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:25:01.280534 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:01.280505 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:25:04.842950 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:04.842913 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:25:05.075345 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.075310 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv"] Jul 31 15:25:05.075737 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.075722 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de18341-e1b9-444e-85e8-b088d067d4c9" containerName="main" Jul 31 15:25:05.075794 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.075738 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de18341-e1b9-444e-85e8-b088d067d4c9" containerName="main" Jul 31 15:25:05.075835 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.075802 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de18341-e1b9-444e-85e8-b088d067d4c9" containerName="main" Jul 31 15:25:05.079058 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.079041 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:25:05.082329 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.082303 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\"" Jul 31 15:25:05.082451 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.082312 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-slkpb\"" Jul 31 15:25:05.082451 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.082313 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\"" Jul 31 15:25:05.082451 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.082313 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\"" Jul 31 15:25:05.082691 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.082338 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-sdx99\"" Jul 31 15:25:05.087657 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.087633 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv"] Jul 31 15:25:05.097947 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.097892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:25:05.250516 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.250483 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsshz\" (UniqueName: \"kubernetes.io/projected/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-kube-api-access-qsshz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv\" (UID: \"041b3e0c-1bd5-4705-9d2a-478577fb1dd5\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:25:05.250708 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.250523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv\" (UID: \"041b3e0c-1bd5-4705-9d2a-478577fb1dd5\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:25:05.352121 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.352032 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsshz\" (UniqueName: \"kubernetes.io/projected/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-kube-api-access-qsshz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv\" (UID: \"041b3e0c-1bd5-4705-9d2a-478577fb1dd5\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:25:05.352121 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.352074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv\" (UID: \"041b3e0c-1bd5-4705-9d2a-478577fb1dd5\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:25:05.354531 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.354508 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv\" (UID: \"041b3e0c-1bd5-4705-9d2a-478577fb1dd5\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:25:05.359970 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.359946 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsshz\" (UniqueName: \"kubernetes.io/projected/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-kube-api-access-qsshz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv\" (UID: \"041b3e0c-1bd5-4705-9d2a-478577fb1dd5\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:25:05.390534 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.390509 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:25:05.523908 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:05.523882 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv"] Jul 31 15:25:05.525709 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:25:05.525685 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041b3e0c_1bd5_4705_9d2a_478577fb1dd5.slice/crio-724f3085ea1a26635697585b4396cdd6c8dd5869e6ad5751bea265d1ccd94c6f WatchSource:0}: Error finding container 724f3085ea1a26635697585b4396cdd6c8dd5869e6ad5751bea265d1ccd94c6f: Status 404 returned error can't find the container with id 724f3085ea1a26635697585b4396cdd6c8dd5869e6ad5751bea265d1ccd94c6f Jul 31 15:25:06.277311 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:06.277282 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" event={"ID":"041b3e0c-1bd5-4705-9d2a-478577fb1dd5","Type":"ContainerStarted","Data":"f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03"} Jul 31 15:25:06.277714 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:06.277331 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" event={"ID":"041b3e0c-1bd5-4705-9d2a-478577fb1dd5","Type":"ContainerStarted","Data":"724f3085ea1a26635697585b4396cdd6c8dd5869e6ad5751bea265d1ccd94c6f"} Jul 31 15:25:06.277714 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:06.277407 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:25:06.298427 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:06.298379 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" podStartSLOduration=1.298366069 podStartE2EDuration="1.298366069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:25:06.297014391 +0000 UTC m=+2217.898450729" watchObservedRunningTime="2026-07-31 15:25:06.298366069 +0000 UTC m=+2217.899802405" Jul 31 15:25:10.118674 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:10.118630 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:25:10.218016 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:10.217984 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:25:10.475551 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:10.475525 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:25:25.469552 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:25.469515 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:25:25.567766 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:25.567722 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:25:37.287983 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:37.287955 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:25:50.512500 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:50.512454 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:25:51.707912 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:25:51.707883 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:26:07.476409 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:26:07.476377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:26:07.576758 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:26:07.576724 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:26:07.683324 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:26:07.683294 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:26:12.550547 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:26:12.550512 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:26:32.990829 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:26:32.990796 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:26:45.699371 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:26:45.699336 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:26:49.395200 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:26:49.395163 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:26:50.055991 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:26:50.055957 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:05.159140 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:05.159020 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:05.258900 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:05.258869 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:18.731672 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:18.731637 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:23.007325 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:23.007296 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:23.014640 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:23.014607 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv"] Jul 31 15:27:23.106000 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:23.105963 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:27.288443 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:27.288404 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" podUID="041b3e0c-1bd5-4705-9d2a-478577fb1dd5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:27:37.288242 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:37.288199 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" podUID="041b3e0c-1bd5-4705-9d2a-478577fb1dd5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:27:38.035307 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:38.035266 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" podUID="041b3e0c-1bd5-4705-9d2a-478577fb1dd5" containerName="main" containerID="cri-o://f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03" gracePeriod=45 Jul 31 15:27:38.287450 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:38.287393 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:27:38.414767 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:38.414739 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-tls-certs\") pod \"041b3e0c-1bd5-4705-9d2a-478577fb1dd5\" (UID: \"041b3e0c-1bd5-4705-9d2a-478577fb1dd5\") " Jul 31 15:27:38.415234 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:38.414784 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsshz\" (UniqueName: \"kubernetes.io/projected/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-kube-api-access-qsshz\") pod \"041b3e0c-1bd5-4705-9d2a-478577fb1dd5\" (UID: \"041b3e0c-1bd5-4705-9d2a-478577fb1dd5\") " Jul 31 15:27:38.417086 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:38.417062 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "041b3e0c-1bd5-4705-9d2a-478577fb1dd5" (UID: "041b3e0c-1bd5-4705-9d2a-478577fb1dd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:27:38.417219 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:38.417195 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-kube-api-access-qsshz" (OuterVolumeSpecName: "kube-api-access-qsshz") pod "041b3e0c-1bd5-4705-9d2a-478577fb1dd5" (UID: "041b3e0c-1bd5-4705-9d2a-478577fb1dd5"). InnerVolumeSpecName "kube-api-access-qsshz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:27:38.515803 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:38.515754 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:27:38.515803 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:38.515804 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsshz\" (UniqueName: \"kubernetes.io/projected/041b3e0c-1bd5-4705-9d2a-478577fb1dd5-kube-api-access-qsshz\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:27:39.111590 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:39.111491 2582 generic.go:356] "Generic (PLEG): container finished" podID="041b3e0c-1bd5-4705-9d2a-478577fb1dd5" containerID="f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03" exitCode=0 Jul 31 15:27:39.111590 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:39.111548 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" Jul 31 15:27:39.111821 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:39.111548 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" event={"ID":"041b3e0c-1bd5-4705-9d2a-478577fb1dd5","Type":"ContainerDied","Data":"f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03"} Jul 31 15:27:39.111821 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:39.111698 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv" event={"ID":"041b3e0c-1bd5-4705-9d2a-478577fb1dd5","Type":"ContainerDied","Data":"724f3085ea1a26635697585b4396cdd6c8dd5869e6ad5751bea265d1ccd94c6f"} Jul 31 15:27:39.111821 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:39.111721 2582 scope.go:117] "RemoveContainer" containerID="f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03" Jul 31 15:27:39.120769 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:39.120751 2582 scope.go:117] "RemoveContainer" containerID="f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03" Jul 31 15:27:39.121010 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:27:39.120994 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03\": container with ID starting with f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03 not found: ID does not exist" containerID="f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03" Jul 31 15:27:39.121058 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:39.121016 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03"} err="failed to get container status \"f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03\": rpc error: code = NotFound desc = could not find container \"f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03\": container with ID starting with f4d94aca43139e116abe7c152135f4891b69cabf27e2d908f6624a981ee45f03 not found: ID does not exist" Jul 31 15:27:39.131875 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:39.131852 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv"] Jul 31 15:27:39.133861 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:39.133841 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheg5bgv"] Jul 31 15:27:40.775824 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:40.775789 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041b3e0c-1bd5-4705-9d2a-478577fb1dd5" path="/var/lib/kubelet/pods/041b3e0c-1bd5-4705-9d2a-478577fb1dd5/volumes" Jul 31 15:27:41.232273 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:41.232243 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:41.325691 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:41.325661 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:47.344464 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:47.344432 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:48.629962 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.629929 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:48.856188 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.856154 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd"] Jul 31 15:27:48.856745 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.856724 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041b3e0c-1bd5-4705-9d2a-478577fb1dd5" containerName="main" Jul 31 15:27:48.856745 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.856746 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="041b3e0c-1bd5-4705-9d2a-478577fb1dd5" containerName="main" Jul 31 15:27:48.856882 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.856853 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="041b3e0c-1bd5-4705-9d2a-478577fb1dd5" containerName="main" Jul 31 15:27:48.860057 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.860037 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:27:48.862421 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.862402 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\"" Jul 31 15:27:48.863276 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.863260 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\"" Jul 31 15:27:48.863385 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.863359 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\"" Jul 31 15:27:48.863455 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.863393 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-wc4nt\"" Jul 31 15:27:48.863455 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.863357 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-nr77j\"" Jul 31 15:27:48.868957 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.868931 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd"] Jul 31 15:27:48.882103 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:48.882044 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:49.003681 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:49.003650 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be173dad-aa74-4a04-b935-043da0fe0867-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd\" (UID: \"be173dad-aa74-4a04-b935-043da0fe0867\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:27:49.003813 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:49.003706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8ndl\" (UniqueName: \"kubernetes.io/projected/be173dad-aa74-4a04-b935-043da0fe0867-kube-api-access-w8ndl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd\" (UID: \"be173dad-aa74-4a04-b935-043da0fe0867\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:27:49.104805 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:49.104774 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be173dad-aa74-4a04-b935-043da0fe0867-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd\" (UID: \"be173dad-aa74-4a04-b935-043da0fe0867\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:27:49.104943 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:49.104811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8ndl\" (UniqueName: \"kubernetes.io/projected/be173dad-aa74-4a04-b935-043da0fe0867-kube-api-access-w8ndl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd\" (UID: \"be173dad-aa74-4a04-b935-043da0fe0867\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:27:49.107383 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:49.107355 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be173dad-aa74-4a04-b935-043da0fe0867-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd\" (UID: \"be173dad-aa74-4a04-b935-043da0fe0867\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:27:49.112245 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:49.112222 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8ndl\" (UniqueName: \"kubernetes.io/projected/be173dad-aa74-4a04-b935-043da0fe0867-kube-api-access-w8ndl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd\" (UID: \"be173dad-aa74-4a04-b935-043da0fe0867\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:27:49.171522 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:49.171457 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:27:49.306510 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:49.306480 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd"] Jul 31 15:27:49.307635 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:27:49.307610 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe173dad_aa74_4a04_b935_043da0fe0867.slice/crio-d61e7c456d470e8379a425c98d107df7fc992f67097d71ba7d8b55a5d9cd5a8e WatchSource:0}: Error finding container d61e7c456d470e8379a425c98d107df7fc992f67097d71ba7d8b55a5d9cd5a8e: Status 404 returned error can't find the container with id d61e7c456d470e8379a425c98d107df7fc992f67097d71ba7d8b55a5d9cd5a8e Jul 31 15:27:50.177096 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:50.177068 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" event={"ID":"be173dad-aa74-4a04-b935-043da0fe0867","Type":"ContainerStarted","Data":"daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef"} Jul 31 15:27:50.177469 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:50.177115 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" event={"ID":"be173dad-aa74-4a04-b935-043da0fe0867","Type":"ContainerStarted","Data":"d61e7c456d470e8379a425c98d107df7fc992f67097d71ba7d8b55a5d9cd5a8e"} Jul 31 15:27:50.177469 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:50.177136 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:27:50.197833 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:50.197791 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" podStartSLOduration=2.197776595 podStartE2EDuration="2.197776595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:27:50.196507632 +0000 UTC m=+2381.797943969" watchObservedRunningTime="2026-07-31 15:27:50.197776595 +0000 UTC m=+2381.799212930" Jul 31 15:27:53.431071 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:53.431038 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:53.531626 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:53.531579 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:27:58.609510 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:27:58.609479 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:11.648217 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:11.648181 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:11.748200 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:11.748166 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:11.850672 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:11.850637 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:14.598638 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:14.598608 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:17.277009 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:17.276971 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:21.183930 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:21.183900 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:28:23.936101 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:23.936055 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:40.252642 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:40.252609 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:44.596005 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:44.595969 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:53.893768 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:53.893740 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:53.993891 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:53.993857 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:54.377670 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:54.377597 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:59.168175 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:59.168140 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:28:59.268542 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:28:59.268515 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:09.763636 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:09.763598 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:09.832182 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:09.832156 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:11.120880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:11.120843 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd"] Jul 31 15:29:11.131051 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:11.131022 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:11.184268 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:11.184230 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" podUID="be173dad-aa74-4a04-b935-043da0fe0867" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:29:11.191673 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:11.191648 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:21.183927 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:21.183888 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" podUID="be173dad-aa74-4a04-b935-043da0fe0867" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:29:22.112444 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:22.112414 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:26.147283 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.147238 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" podUID="be173dad-aa74-4a04-b935-043da0fe0867" containerName="main" containerID="cri-o://daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef" gracePeriod=45 Jul 31 15:29:26.398668 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.398613 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:29:26.427826 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.427801 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be173dad-aa74-4a04-b935-043da0fe0867-tls-certs\") pod \"be173dad-aa74-4a04-b935-043da0fe0867\" (UID: \"be173dad-aa74-4a04-b935-043da0fe0867\") " Jul 31 15:29:26.427975 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.427896 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8ndl\" (UniqueName: \"kubernetes.io/projected/be173dad-aa74-4a04-b935-043da0fe0867-kube-api-access-w8ndl\") pod \"be173dad-aa74-4a04-b935-043da0fe0867\" (UID: \"be173dad-aa74-4a04-b935-043da0fe0867\") " Jul 31 15:29:26.430063 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.430035 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be173dad-aa74-4a04-b935-043da0fe0867-kube-api-access-w8ndl" (OuterVolumeSpecName: "kube-api-access-w8ndl") pod "be173dad-aa74-4a04-b935-043da0fe0867" (UID: "be173dad-aa74-4a04-b935-043da0fe0867"). InnerVolumeSpecName "kube-api-access-w8ndl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:29:26.430171 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.430060 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be173dad-aa74-4a04-b935-043da0fe0867-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be173dad-aa74-4a04-b935-043da0fe0867" (UID: "be173dad-aa74-4a04-b935-043da0fe0867"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:29:26.529145 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.529118 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8ndl\" (UniqueName: \"kubernetes.io/projected/be173dad-aa74-4a04-b935-043da0fe0867-kube-api-access-w8ndl\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:29:26.529145 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.529146 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be173dad-aa74-4a04-b935-043da0fe0867-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:29:26.684225 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.684140 2582 generic.go:356] "Generic (PLEG): container finished" podID="be173dad-aa74-4a04-b935-043da0fe0867" containerID="daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef" exitCode=0 Jul 31 15:29:26.684225 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.684192 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" Jul 31 15:29:26.684398 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.684231 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" event={"ID":"be173dad-aa74-4a04-b935-043da0fe0867","Type":"ContainerDied","Data":"daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef"} Jul 31 15:29:26.684398 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.684313 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd" event={"ID":"be173dad-aa74-4a04-b935-043da0fe0867","Type":"ContainerDied","Data":"d61e7c456d470e8379a425c98d107df7fc992f67097d71ba7d8b55a5d9cd5a8e"} Jul 31 15:29:26.684398 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.684340 2582 scope.go:117] "RemoveContainer" containerID="daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef" Jul 31 15:29:26.694270 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.694258 2582 scope.go:117] "RemoveContainer" containerID="daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef" Jul 31 15:29:26.694515 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:29:26.694497 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef\": container with ID starting with daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef not found: ID does not exist" containerID="daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef" Jul 31 15:29:26.694608 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.694519 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef"} err="failed to get container status \"daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef\": rpc error: code = NotFound desc = could not find container \"daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef\": container with ID starting with daea101ba660550f5ea3e421ee11fb8d5239e85ba1effb88fe96d8f1af5eb4ef not found: ID does not exist" Jul 31 15:29:26.707473 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.707447 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd"] Jul 31 15:29:26.713532 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.713513 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh74nd"] Jul 31 15:29:26.775363 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:26.775338 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be173dad-aa74-4a04-b935-043da0fe0867" path="/var/lib/kubelet/pods/be173dad-aa74-4a04-b935-043da0fe0867/volumes" Jul 31 15:29:30.660346 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:30.660308 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:30.761049 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:30.761019 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:32.891954 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:32.891926 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:32.994388 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:32.994361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:33.091302 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:33.091272 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:36.983816 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:36.983784 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:39.005566 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:39.005523 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:39.106748 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:39.106720 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:43.071209 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:43.071173 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:43.172126 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:43.172098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:43.807790 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:43.807763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:43.913867 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:43.913839 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:44.010900 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:44.010864 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:44.204822 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:44.204793 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:44.300210 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:44.300178 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:48.569747 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:48.569709 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:48.887911 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:48.887877 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:55.713137 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.713097 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:55.957632 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.957602 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5"] Jul 31 15:29:55.958000 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.957987 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be173dad-aa74-4a04-b935-043da0fe0867" containerName="main" Jul 31 15:29:55.958057 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.958001 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="be173dad-aa74-4a04-b935-043da0fe0867" containerName="main" Jul 31 15:29:55.958100 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.958061 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="be173dad-aa74-4a04-b935-043da0fe0867" containerName="main" Jul 31 15:29:55.961001 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.960983 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:29:55.964352 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.964300 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\"" Jul 31 15:29:55.964949 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.964927 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-hzhdm\"" Jul 31 15:29:55.965057 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.964925 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\"" Jul 31 15:29:55.965057 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.965035 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\"" Jul 31 15:29:55.965359 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.965061 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-mkqmv\"" Jul 31 15:29:55.970092 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.970073 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5"] Jul 31 15:29:55.976539 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:55.976517 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:29:56.082507 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.082478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b5febad-1572-436b-9e5f-69d3fdcc71b7-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5\" (UID: \"8b5febad-1572-436b-9e5f-69d3fdcc71b7\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:29:56.082684 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.082662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxrf\" (UniqueName: \"kubernetes.io/projected/8b5febad-1572-436b-9e5f-69d3fdcc71b7-kube-api-access-hdxrf\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5\" (UID: \"8b5febad-1572-436b-9e5f-69d3fdcc71b7\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:29:56.183634 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.183607 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxrf\" (UniqueName: \"kubernetes.io/projected/8b5febad-1572-436b-9e5f-69d3fdcc71b7-kube-api-access-hdxrf\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5\" (UID: \"8b5febad-1572-436b-9e5f-69d3fdcc71b7\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:29:56.183758 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.183643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b5febad-1572-436b-9e5f-69d3fdcc71b7-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5\" (UID: \"8b5febad-1572-436b-9e5f-69d3fdcc71b7\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:29:56.186051 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.186030 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b5febad-1572-436b-9e5f-69d3fdcc71b7-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5\" (UID: \"8b5febad-1572-436b-9e5f-69d3fdcc71b7\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:29:56.193069 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.193048 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxrf\" (UniqueName: \"kubernetes.io/projected/8b5febad-1572-436b-9e5f-69d3fdcc71b7-kube-api-access-hdxrf\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5\" (UID: \"8b5febad-1572-436b-9e5f-69d3fdcc71b7\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:29:56.271816 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.271767 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:29:56.406928 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.406904 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5"] Jul 31 15:29:56.408481 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:29:56.408452 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5febad_1572_436b_9e5f_69d3fdcc71b7.slice/crio-18bb151fb58df32315e75b005000e8ff32f51cc99e44b46ae1317e670d9b5ec0 WatchSource:0}: Error finding container 18bb151fb58df32315e75b005000e8ff32f51cc99e44b46ae1317e670d9b5ec0: Status 404 returned error can't find the container with id 18bb151fb58df32315e75b005000e8ff32f51cc99e44b46ae1317e670d9b5ec0 Jul 31 15:29:56.841809 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.841768 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" event={"ID":"8b5febad-1572-436b-9e5f-69d3fdcc71b7","Type":"ContainerStarted","Data":"261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156"} Jul 31 15:29:56.842213 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.841836 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" event={"ID":"8b5febad-1572-436b-9e5f-69d3fdcc71b7","Type":"ContainerStarted","Data":"18bb151fb58df32315e75b005000e8ff32f51cc99e44b46ae1317e670d9b5ec0"} Jul 31 15:29:56.842213 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.841882 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:29:56.859292 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:29:56.859245 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" podStartSLOduration=1.8592319769999999 podStartE2EDuration="1.859231977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:29:56.858157194 +0000 UTC m=+2508.459593556" watchObservedRunningTime="2026-07-31 15:29:56.859231977 +0000 UTC m=+2508.460668314" Jul 31 15:30:25.597102 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:25.597068 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:27.848703 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:27.848671 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:30:28.943496 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:28.943468 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:39.033014 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:39.032971 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:41.098395 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:41.098362 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5"] Jul 31 15:30:41.112921 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:41.112868 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:41.200349 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:41.200321 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:42.703931 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:42.703899 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:43.627251 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:43.627222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:43.727710 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:43.727674 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:43.832801 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:43.832777 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:47.848719 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:47.848683 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" podUID="8b5febad-1572-436b-9e5f-69d3fdcc71b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:30:48.047652 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:48.047625 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:48.147680 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:48.147648 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:48.263126 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:48.263099 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:48.700712 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:48.700681 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:48.808702 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:48.808676 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:52.932965 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:52.932932 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:53.198846 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:53.198771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:53.308043 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:53.308017 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:55.033339 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:55.033309 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:55.385911 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:55.385880 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:56.120232 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:56.120193 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" podUID="8b5febad-1572-436b-9e5f-69d3fdcc71b7" containerName="main" containerID="cri-o://261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156" gracePeriod=45 Jul 31 15:30:56.372433 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:56.372380 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:30:56.395422 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:56.395397 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b5febad-1572-436b-9e5f-69d3fdcc71b7-tls-certs\") pod \"8b5febad-1572-436b-9e5f-69d3fdcc71b7\" (UID: \"8b5febad-1572-436b-9e5f-69d3fdcc71b7\") " Jul 31 15:30:56.395539 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:56.395464 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdxrf\" (UniqueName: \"kubernetes.io/projected/8b5febad-1572-436b-9e5f-69d3fdcc71b7-kube-api-access-hdxrf\") pod \"8b5febad-1572-436b-9e5f-69d3fdcc71b7\" (UID: \"8b5febad-1572-436b-9e5f-69d3fdcc71b7\") " Jul 31 15:30:56.397630 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:56.397604 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b5febad-1572-436b-9e5f-69d3fdcc71b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b5febad-1572-436b-9e5f-69d3fdcc71b7" (UID: "8b5febad-1572-436b-9e5f-69d3fdcc71b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:30:56.397758 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:56.397734 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5febad-1572-436b-9e5f-69d3fdcc71b7-kube-api-access-hdxrf" (OuterVolumeSpecName: "kube-api-access-hdxrf") pod "8b5febad-1572-436b-9e5f-69d3fdcc71b7" (UID: "8b5febad-1572-436b-9e5f-69d3fdcc71b7"). InnerVolumeSpecName "kube-api-access-hdxrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:30:56.496182 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:56.496155 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b5febad-1572-436b-9e5f-69d3fdcc71b7-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:30:56.496182 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:56.496180 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxrf\" (UniqueName: \"kubernetes.io/projected/8b5febad-1572-436b-9e5f-69d3fdcc71b7-kube-api-access-hdxrf\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:30:57.159073 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:57.159040 2582 generic.go:356] "Generic (PLEG): container finished" podID="8b5febad-1572-436b-9e5f-69d3fdcc71b7" containerID="261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156" exitCode=0 Jul 31 15:30:57.159501 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:57.159088 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" Jul 31 15:30:57.159501 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:57.159122 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" event={"ID":"8b5febad-1572-436b-9e5f-69d3fdcc71b7","Type":"ContainerDied","Data":"261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156"} Jul 31 15:30:57.159501 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:57.159176 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5" event={"ID":"8b5febad-1572-436b-9e5f-69d3fdcc71b7","Type":"ContainerDied","Data":"18bb151fb58df32315e75b005000e8ff32f51cc99e44b46ae1317e670d9b5ec0"} Jul 31 15:30:57.159501 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:57.159199 2582 scope.go:117] "RemoveContainer" containerID="261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156" Jul 31 15:30:57.168041 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:57.168030 2582 scope.go:117] "RemoveContainer" containerID="261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156" Jul 31 15:30:57.168294 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:30:57.168277 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156\": container with ID starting with 261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156 not found: ID does not exist" containerID="261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156" Jul 31 15:30:57.168344 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:57.168300 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156"} err="failed to get container status \"261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156\": rpc error: code = NotFound desc = could not find container \"261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156\": container with ID starting with 261c2b1cea93b81dad1d594cb09b5b1576598a1d290ce14220540e65a6cb2156 not found: ID does not exist" Jul 31 15:30:57.176541 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:57.176520 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5"] Jul 31 15:30:57.182841 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:57.182822 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-pb6d5"] Jul 31 15:30:58.423849 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:58.423800 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:58.519338 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:58.519304 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:58.775849 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:58.775777 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b5febad-1572-436b-9e5f-69d3fdcc71b7" path="/var/lib/kubelet/pods/8b5febad-1572-436b-9e5f-69d3fdcc71b7/volumes" Jul 31 15:30:59.707853 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:59.707823 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:30:59.808909 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:30:59.808880 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:31:04.385217 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:04.385166 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:31:04.485661 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:04.485629 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:31:06.041130 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.041095 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:31:06.223382 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.223348 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc"] Jul 31 15:31:06.223954 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.223935 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5febad-1572-436b-9e5f-69d3fdcc71b7" containerName="main" Jul 31 15:31:06.224024 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.223959 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5febad-1572-436b-9e5f-69d3fdcc71b7" containerName="main" Jul 31 15:31:06.224088 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.224075 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5febad-1572-436b-9e5f-69d3fdcc71b7" containerName="main" Jul 31 15:31:06.228714 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.228696 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:31:06.231100 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.231072 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\"" Jul 31 15:31:06.232065 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.232049 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-72686\"" Jul 31 15:31:06.232127 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.232049 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\"" Jul 31 15:31:06.232127 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.232114 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\"" Jul 31 15:31:06.232211 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.232160 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-xpppn\"" Jul 31 15:31:06.236202 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.236082 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc"] Jul 31 15:31:06.277873 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.277842 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:31:06.278426 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.278402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt4sm\" (UniqueName: \"kubernetes.io/projected/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-kube-api-access-wt4sm\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc\" (UID: \"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:31:06.278574 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.278508 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc\" (UID: \"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:31:06.379280 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.379195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc\" (UID: \"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:31:06.379280 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.379239 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt4sm\" (UniqueName: \"kubernetes.io/projected/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-kube-api-access-wt4sm\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc\" (UID: \"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:31:06.381974 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.381950 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc\" (UID: \"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:31:06.388761 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.388739 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt4sm\" (UniqueName: \"kubernetes.io/projected/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-kube-api-access-wt4sm\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc\" (UID: \"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:31:06.541159 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.541128 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:31:06.680480 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:06.680457 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc"] Jul 31 15:31:06.682689 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:31:06.682663 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b4dbcc_2b3e_4c4b_aefe_5e52d30d07b5.slice/crio-188a8411bade2381b1ceb661e666d58d75589bc6c7b2db6f4e679640a7f44aa6 WatchSource:0}: Error finding container 188a8411bade2381b1ceb661e666d58d75589bc6c7b2db6f4e679640a7f44aa6: Status 404 returned error can't find the container with id 188a8411bade2381b1ceb661e666d58d75589bc6c7b2db6f4e679640a7f44aa6 Jul 31 15:31:07.221026 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:07.220994 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" event={"ID":"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5","Type":"ContainerStarted","Data":"bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023"} Jul 31 15:31:07.221445 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:07.221046 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" event={"ID":"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5","Type":"ContainerStarted","Data":"188a8411bade2381b1ceb661e666d58d75589bc6c7b2db6f4e679640a7f44aa6"} Jul 31 15:31:07.221445 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:07.221125 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:31:07.242849 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:07.242792 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" podStartSLOduration=1.24277749 podStartE2EDuration="1.24277749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:31:07.241244046 +0000 UTC m=+2578.842680383" watchObservedRunningTime="2026-07-31 15:31:07.24277749 +0000 UTC m=+2578.844213827" Jul 31 15:31:26.644206 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:26.644166 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:31:28.253196 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:28.253167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:31:29.645536 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:29.645506 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:31:38.227836 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:31:38.227805 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:32:00.222002 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:00.221956 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:05.023296 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:05.023265 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:05.147676 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:05.147643 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:05.239205 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:05.239178 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:09.254867 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.254835 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:09.510991 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.510901 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq"] Jul 31 15:32:09.514766 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.514735 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:32:09.517538 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.517519 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\"" Jul 31 15:32:09.518654 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.518635 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\"" Jul 31 15:32:09.518777 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.518717 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\"" Jul 31 15:32:09.518777 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.518737 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-s2fls\"" Jul 31 15:32:09.518908 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.518668 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-9sdq9\"" Jul 31 15:32:09.528752 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.528729 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq"] Jul 31 15:32:09.530049 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.530025 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:09.591210 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.591174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2d1daf-662f-441a-83d9-48d65608d5b2-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq\" (UID: \"8f2d1daf-662f-441a-83d9-48d65608d5b2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:32:09.591210 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.591211 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpt2g\" (UniqueName: \"kubernetes.io/projected/8f2d1daf-662f-441a-83d9-48d65608d5b2-kube-api-access-tpt2g\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq\" (UID: \"8f2d1daf-662f-441a-83d9-48d65608d5b2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:32:09.693327 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.693255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2d1daf-662f-441a-83d9-48d65608d5b2-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq\" (UID: \"8f2d1daf-662f-441a-83d9-48d65608d5b2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:32:09.693327 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.693320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpt2g\" (UniqueName: \"kubernetes.io/projected/8f2d1daf-662f-441a-83d9-48d65608d5b2-kube-api-access-tpt2g\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq\" (UID: \"8f2d1daf-662f-441a-83d9-48d65608d5b2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:32:09.696365 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.696336 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2d1daf-662f-441a-83d9-48d65608d5b2-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq\" (UID: \"8f2d1daf-662f-441a-83d9-48d65608d5b2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:32:09.721442 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.721413 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpt2g\" (UniqueName: \"kubernetes.io/projected/8f2d1daf-662f-441a-83d9-48d65608d5b2-kube-api-access-tpt2g\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq\" (UID: \"8f2d1daf-662f-441a-83d9-48d65608d5b2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:32:09.831693 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.831605 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:32:09.980890 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:09.980863 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq"] Jul 31 15:32:09.982553 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:32:09.982529 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2d1daf_662f_441a_83d9_48d65608d5b2.slice/crio-31352eea25831a3fd143a95d11cdb067693664bf30c19eca3ebca6d3cfcee96a WatchSource:0}: Error finding container 31352eea25831a3fd143a95d11cdb067693664bf30c19eca3ebca6d3cfcee96a: Status 404 returned error can't find the container with id 31352eea25831a3fd143a95d11cdb067693664bf30c19eca3ebca6d3cfcee96a Jul 31 15:32:10.066448 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:10.066419 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:10.555920 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:10.555883 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" event={"ID":"8f2d1daf-662f-441a-83d9-48d65608d5b2","Type":"ContainerStarted","Data":"0aa2e378aaf2cb93871573de49a89fc2ad3617c866a8e246d460604ce9e099cc"} Jul 31 15:32:10.556378 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:10.555933 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" event={"ID":"8f2d1daf-662f-441a-83d9-48d65608d5b2","Type":"ContainerStarted","Data":"31352eea25831a3fd143a95d11cdb067693664bf30c19eca3ebca6d3cfcee96a"} Jul 31 15:32:10.556378 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:10.555972 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:32:10.591515 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:10.591463 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" podStartSLOduration=1.591448623 podStartE2EDuration="1.591448623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:32:10.588722973 +0000 UTC m=+2642.190159311" watchObservedRunningTime="2026-07-31 15:32:10.591448623 +0000 UTC m=+2642.192884960" Jul 31 15:32:23.166934 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:23.166901 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:31.413298 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:31.413253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:32.862695 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:32.862658 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:38.592974 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:38.592940 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:41.563768 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:41.563735 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:32:47.696312 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:47.696268 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:54.122191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:54.122157 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:32:54.221880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:32:54.221845 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:33:07.513766 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:07.513674 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:33:07.515066 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:07.515043 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc"] Jul 31 15:33:07.608985 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:07.608938 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:33:07.709384 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:07.709350 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:33:08.228314 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:08.228272 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" podUID="25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:33:18.228139 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:18.228090 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" podUID="25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:33:22.535611 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.535565 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" podUID="25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5" containerName="main" containerID="cri-o://bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023" gracePeriod=45 Jul 31 15:33:22.793914 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.793850 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:33:22.834809 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.834779 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-tls-certs\") pod \"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5\" (UID: \"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5\") " Jul 31 15:33:22.834983 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.834842 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt4sm\" (UniqueName: \"kubernetes.io/projected/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-kube-api-access-wt4sm\") pod \"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5\" (UID: \"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5\") " Jul 31 15:33:22.837085 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.837051 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5" (UID: "25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:33:22.837202 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.837101 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-kube-api-access-wt4sm" (OuterVolumeSpecName: "kube-api-access-wt4sm") pod "25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5" (UID: "25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5"). InnerVolumeSpecName "kube-api-access-wt4sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:33:22.935501 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.935464 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:33:22.935501 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.935494 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt4sm\" (UniqueName: \"kubernetes.io/projected/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5-kube-api-access-wt4sm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:33:22.943404 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.943370 2582 generic.go:356] "Generic (PLEG): container finished" podID="25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5" containerID="bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023" exitCode=0 Jul 31 15:33:22.943577 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.943413 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" Jul 31 15:33:22.943577 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.943458 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" event={"ID":"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5","Type":"ContainerDied","Data":"bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023"} Jul 31 15:33:22.943577 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.943516 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc" event={"ID":"25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5","Type":"ContainerDied","Data":"188a8411bade2381b1ceb661e666d58d75589bc6c7b2db6f4e679640a7f44aa6"} Jul 31 15:33:22.943577 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.943548 2582 scope.go:117] "RemoveContainer" containerID="bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023" Jul 31 15:33:22.953792 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.953765 2582 scope.go:117] "RemoveContainer" containerID="bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023" Jul 31 15:33:22.954070 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:33:22.954045 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023\": container with ID starting with bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023 not found: ID does not exist" containerID="bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023" Jul 31 15:33:22.954166 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.954077 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023"} err="failed to get container status \"bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023\": rpc error: code = NotFound desc = could not find container \"bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023\": container with ID starting with bbd3638859e2413dfc2f336655a490a5e003cfe54d3e3916027b1e03e6d52023 not found: ID does not exist" Jul 31 15:33:22.966476 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.966445 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc"] Jul 31 15:33:22.972357 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:22.972332 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-r6rsc"] Jul 31 15:33:24.776863 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:24.776832 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5" path="/var/lib/kubelet/pods/25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5/volumes" Jul 31 15:33:28.554696 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:28.554663 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:33:28.728293 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:28.728256 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:33:42.209435 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:33:42.209405 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:04.628144 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:04.628112 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:06.080054 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:06.080023 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:14.095217 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:14.095183 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:14.104676 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:14.104650 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq"] Jul 31 15:34:14.153463 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:14.153437 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:21.564181 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:21.564144 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" podUID="8f2d1daf-662f-441a-83d9-48d65608d5b2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:34:29.124405 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:29.124346 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" podUID="8f2d1daf-662f-441a-83d9-48d65608d5b2" containerName="main" containerID="cri-o://0aa2e378aaf2cb93871573de49a89fc2ad3617c866a8e246d460604ce9e099cc" gracePeriod=45 Jul 31 15:34:29.326692 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:29.326657 2582 generic.go:356] "Generic (PLEG): container finished" podID="8f2d1daf-662f-441a-83d9-48d65608d5b2" containerID="0aa2e378aaf2cb93871573de49a89fc2ad3617c866a8e246d460604ce9e099cc" exitCode=0 Jul 31 15:34:29.326840 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:29.326729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" event={"ID":"8f2d1daf-662f-441a-83d9-48d65608d5b2","Type":"ContainerDied","Data":"0aa2e378aaf2cb93871573de49a89fc2ad3617c866a8e246d460604ce9e099cc"} Jul 31 15:34:29.376610 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:29.376540 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:34:29.498196 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:29.498169 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2d1daf-662f-441a-83d9-48d65608d5b2-tls-certs\") pod \"8f2d1daf-662f-441a-83d9-48d65608d5b2\" (UID: \"8f2d1daf-662f-441a-83d9-48d65608d5b2\") " Jul 31 15:34:29.498316 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:29.498256 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpt2g\" (UniqueName: \"kubernetes.io/projected/8f2d1daf-662f-441a-83d9-48d65608d5b2-kube-api-access-tpt2g\") pod \"8f2d1daf-662f-441a-83d9-48d65608d5b2\" (UID: \"8f2d1daf-662f-441a-83d9-48d65608d5b2\") " Jul 31 15:34:29.500239 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:29.500211 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2d1daf-662f-441a-83d9-48d65608d5b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f2d1daf-662f-441a-83d9-48d65608d5b2" (UID: "8f2d1daf-662f-441a-83d9-48d65608d5b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:34:29.500335 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:29.500246 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2d1daf-662f-441a-83d9-48d65608d5b2-kube-api-access-tpt2g" (OuterVolumeSpecName: "kube-api-access-tpt2g") pod "8f2d1daf-662f-441a-83d9-48d65608d5b2" (UID: "8f2d1daf-662f-441a-83d9-48d65608d5b2"). InnerVolumeSpecName "kube-api-access-tpt2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:34:29.599637 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:29.599614 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpt2g\" (UniqueName: \"kubernetes.io/projected/8f2d1daf-662f-441a-83d9-48d65608d5b2-kube-api-access-tpt2g\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:34:29.599637 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:29.599634 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2d1daf-662f-441a-83d9-48d65608d5b2-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:34:30.333262 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:30.333236 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" Jul 31 15:34:30.333262 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:30.333239 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq" event={"ID":"8f2d1daf-662f-441a-83d9-48d65608d5b2","Type":"ContainerDied","Data":"31352eea25831a3fd143a95d11cdb067693664bf30c19eca3ebca6d3cfcee96a"} Jul 31 15:34:30.333748 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:30.333300 2582 scope.go:117] "RemoveContainer" containerID="0aa2e378aaf2cb93871573de49a89fc2ad3617c866a8e246d460604ce9e099cc" Jul 31 15:34:30.360123 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:30.360094 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq"] Jul 31 15:34:30.368680 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:30.368655 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemmjnq"] Jul 31 15:34:30.776897 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:30.776871 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f2d1daf-662f-441a-83d9-48d65608d5b2" path="/var/lib/kubelet/pods/8f2d1daf-662f-441a-83d9-48d65608d5b2/volumes" Jul 31 15:34:34.125856 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:34.125800 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:34.221640 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:34.221609 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:38.258174 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.258144 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:38.430998 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.430960 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9"] Jul 31 15:34:38.431552 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.431532 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2d1daf-662f-441a-83d9-48d65608d5b2" containerName="main" Jul 31 15:34:38.431552 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.431572 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2d1daf-662f-441a-83d9-48d65608d5b2" containerName="main" Jul 31 15:34:38.431698 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.431597 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5" containerName="main" Jul 31 15:34:38.431698 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.431606 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5" containerName="main" Jul 31 15:34:38.431698 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.431691 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="25b4dbcc-2b3e-4c4b-aefe-5e52d30d07b5" containerName="main" Jul 31 15:34:38.431822 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.431702 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2d1daf-662f-441a-83d9-48d65608d5b2" containerName="main" Jul 31 15:34:38.436355 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.436334 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:34:38.440095 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.439722 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-zt6n9\"" Jul 31 15:34:38.440095 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.439728 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\"" Jul 31 15:34:38.440095 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.439789 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\"" Jul 31 15:34:38.440095 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.439729 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-5j4vn\"" Jul 31 15:34:38.440095 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.440094 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\"" Jul 31 15:34:38.445177 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.445155 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9"] Jul 31 15:34:38.468904 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.468883 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7281bcf1-2640-4197-81ac-18cea43dee45-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9\" (UID: \"7281bcf1-2640-4197-81ac-18cea43dee45\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:34:38.469009 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.468930 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qzzp\" (UniqueName: \"kubernetes.io/projected/7281bcf1-2640-4197-81ac-18cea43dee45-kube-api-access-2qzzp\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9\" (UID: \"7281bcf1-2640-4197-81ac-18cea43dee45\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:34:38.570163 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.570103 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7281bcf1-2640-4197-81ac-18cea43dee45-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9\" (UID: \"7281bcf1-2640-4197-81ac-18cea43dee45\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:34:38.570265 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.570159 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qzzp\" (UniqueName: \"kubernetes.io/projected/7281bcf1-2640-4197-81ac-18cea43dee45-kube-api-access-2qzzp\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9\" (UID: \"7281bcf1-2640-4197-81ac-18cea43dee45\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:34:38.572862 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.572840 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7281bcf1-2640-4197-81ac-18cea43dee45-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9\" (UID: \"7281bcf1-2640-4197-81ac-18cea43dee45\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:34:38.578570 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.578537 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qzzp\" (UniqueName: \"kubernetes.io/projected/7281bcf1-2640-4197-81ac-18cea43dee45-kube-api-access-2qzzp\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9\" (UID: \"7281bcf1-2640-4197-81ac-18cea43dee45\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:34:38.750159 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.750133 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:34:38.881537 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:38.881513 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9"] Jul 31 15:34:38.883427 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:34:38.883405 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7281bcf1_2640_4197_81ac_18cea43dee45.slice/crio-7ebeb87e33e876e9a27b73407540c0a47a7e0d75b9ab929fb03de9875fa0993e WatchSource:0}: Error finding container 7ebeb87e33e876e9a27b73407540c0a47a7e0d75b9ab929fb03de9875fa0993e: Status 404 returned error can't find the container with id 7ebeb87e33e876e9a27b73407540c0a47a7e0d75b9ab929fb03de9875fa0993e Jul 31 15:34:39.380978 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:39.380949 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" event={"ID":"7281bcf1-2640-4197-81ac-18cea43dee45","Type":"ContainerStarted","Data":"efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423"} Jul 31 15:34:39.381377 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:39.380995 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" event={"ID":"7281bcf1-2640-4197-81ac-18cea43dee45","Type":"ContainerStarted","Data":"7ebeb87e33e876e9a27b73407540c0a47a7e0d75b9ab929fb03de9875fa0993e"} Jul 31 15:34:39.381377 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:39.381091 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:34:39.400880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:39.400827 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" podStartSLOduration=1.40081547 podStartE2EDuration="1.40081547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:34:39.3983767 +0000 UTC m=+2790.999813036" watchObservedRunningTime="2026-07-31 15:34:39.40081547 +0000 UTC m=+2791.002251809" Jul 31 15:34:41.642821 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:41.642784 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:46.463476 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:46.463442 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:46.567868 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:46.567842 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:52.084448 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:52.084406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:34:55.919971 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:34:55.919930 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:35:10.388838 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:35:10.388809 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:35:29.888796 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:35:29.888757 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:35:29.954448 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:35:29.954416 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:35:49.952730 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:35:49.952687 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:35:50.053054 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:35:50.053022 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:35:50.153470 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:35:50.153433 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:35:56.368124 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:35:56.368088 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:36:02.380011 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:02.379925 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:36:02.441605 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:02.441535 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:36:08.232982 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:08.232946 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:36:43.412319 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:43.412278 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9"] Jul 31 15:36:43.413407 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:43.413377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:36:43.507505 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:43.507476 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:36:50.389216 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:50.389176 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" podUID="7281bcf1-2640-4197-81ac-18cea43dee45" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:36:58.432303 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:58.432261 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" podUID="7281bcf1-2640-4197-81ac-18cea43dee45" containerName="main" containerID="cri-o://efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423" gracePeriod=45 Jul 31 15:36:58.680143 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:58.680120 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:36:58.789645 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:58.789579 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qzzp\" (UniqueName: \"kubernetes.io/projected/7281bcf1-2640-4197-81ac-18cea43dee45-kube-api-access-2qzzp\") pod \"7281bcf1-2640-4197-81ac-18cea43dee45\" (UID: \"7281bcf1-2640-4197-81ac-18cea43dee45\") " Jul 31 15:36:58.789774 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:58.789676 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7281bcf1-2640-4197-81ac-18cea43dee45-tls-certs\") pod \"7281bcf1-2640-4197-81ac-18cea43dee45\" (UID: \"7281bcf1-2640-4197-81ac-18cea43dee45\") " Jul 31 15:36:58.791674 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:58.791644 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7281bcf1-2640-4197-81ac-18cea43dee45-kube-api-access-2qzzp" (OuterVolumeSpecName: "kube-api-access-2qzzp") pod "7281bcf1-2640-4197-81ac-18cea43dee45" (UID: "7281bcf1-2640-4197-81ac-18cea43dee45"). InnerVolumeSpecName "kube-api-access-2qzzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:36:58.791780 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:58.791680 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7281bcf1-2640-4197-81ac-18cea43dee45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7281bcf1-2640-4197-81ac-18cea43dee45" (UID: "7281bcf1-2640-4197-81ac-18cea43dee45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:36:58.890321 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:58.890302 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7281bcf1-2640-4197-81ac-18cea43dee45-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:36:58.890321 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:58.890319 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qzzp\" (UniqueName: \"kubernetes.io/projected/7281bcf1-2640-4197-81ac-18cea43dee45-kube-api-access-2qzzp\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:36:59.117814 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:59.117742 2582 generic.go:356] "Generic (PLEG): container finished" podID="7281bcf1-2640-4197-81ac-18cea43dee45" containerID="efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423" exitCode=0 Jul 31 15:36:59.117814 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:59.117791 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" Jul 31 15:36:59.117969 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:59.117784 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" event={"ID":"7281bcf1-2640-4197-81ac-18cea43dee45","Type":"ContainerDied","Data":"efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423"} Jul 31 15:36:59.117969 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:59.117910 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9" event={"ID":"7281bcf1-2640-4197-81ac-18cea43dee45","Type":"ContainerDied","Data":"7ebeb87e33e876e9a27b73407540c0a47a7e0d75b9ab929fb03de9875fa0993e"} Jul 31 15:36:59.117969 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:59.117943 2582 scope.go:117] "RemoveContainer" containerID="efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423" Jul 31 15:36:59.127225 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:59.127209 2582 scope.go:117] "RemoveContainer" containerID="efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423" Jul 31 15:36:59.127460 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:36:59.127440 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423\": container with ID starting with efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423 not found: ID does not exist" containerID="efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423" Jul 31 15:36:59.127510 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:59.127467 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423"} err="failed to get container status \"efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423\": rpc error: code = NotFound desc = could not find container \"efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423\": container with ID starting with efd022b0a8bfbd60adfd95d0b0b07241a377687058e9530f22643c335db1d423 not found: ID does not exist" Jul 31 15:36:59.139764 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:59.139744 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9"] Jul 31 15:36:59.145960 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:36:59.145941 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892nkc9"] Jul 31 15:37:00.776811 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:00.776774 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7281bcf1-2640-4197-81ac-18cea43dee45" path="/var/lib/kubelet/pods/7281bcf1-2640-4197-81ac-18cea43dee45/volumes" Jul 31 15:37:03.429265 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:03.429234 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:03.525716 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:03.525685 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:03.671060 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:03.671033 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:03.775321 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:03.775236 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:06.840128 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:06.840095 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:07.627477 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:07.627445 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:10.008513 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:10.008475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:16.000286 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:16.000250 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:16.098216 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:16.098186 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:25.219388 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.219351 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:25.579829 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.579743 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh"] Jul 31 15:37:25.580368 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.580345 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7281bcf1-2640-4197-81ac-18cea43dee45" containerName="main" Jul 31 15:37:25.580368 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.580367 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7281bcf1-2640-4197-81ac-18cea43dee45" containerName="main" Jul 31 15:37:25.580605 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.580465 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7281bcf1-2640-4197-81ac-18cea43dee45" containerName="main" Jul 31 15:37:25.583667 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.583646 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:37:25.587018 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.586993 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-8nk2s\"" Jul 31 15:37:25.587154 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.586993 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-l8frj\"" Jul 31 15:37:25.587154 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.587028 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\"" Jul 31 15:37:25.587154 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.587005 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\"" Jul 31 15:37:25.587154 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.587100 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\"" Jul 31 15:37:25.594052 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.594032 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh"] Jul 31 15:37:25.706450 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.706419 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:25.717664 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.717640 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh\" (UID: \"70cd2b93-b97c-41ff-81f4-b3aa89f2347c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:37:25.717787 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.717678 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpb4d\" (UniqueName: \"kubernetes.io/projected/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-kube-api-access-lpb4d\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh\" (UID: \"70cd2b93-b97c-41ff-81f4-b3aa89f2347c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:37:25.774441 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.774412 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:37:25.819192 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.819159 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh\" (UID: \"70cd2b93-b97c-41ff-81f4-b3aa89f2347c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:37:25.819192 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.819197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpb4d\" (UniqueName: \"kubernetes.io/projected/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-kube-api-access-lpb4d\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh\" (UID: \"70cd2b93-b97c-41ff-81f4-b3aa89f2347c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:37:25.821831 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.821806 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh\" (UID: \"70cd2b93-b97c-41ff-81f4-b3aa89f2347c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:37:25.827282 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.827262 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpb4d\" (UniqueName: \"kubernetes.io/projected/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-kube-api-access-lpb4d\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh\" (UID: \"70cd2b93-b97c-41ff-81f4-b3aa89f2347c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:37:25.895363 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:25.895334 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:37:26.028743 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:26.028715 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh"] Jul 31 15:37:26.030152 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:37:26.030121 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cd2b93_b97c_41ff_81f4_b3aa89f2347c.slice/crio-a8f6fdca0d516a02c59376ad2f680907766e6ebd6fbb5ff9841116e7b9396907 WatchSource:0}: Error finding container a8f6fdca0d516a02c59376ad2f680907766e6ebd6fbb5ff9841116e7b9396907: Status 404 returned error can't find the container with id a8f6fdca0d516a02c59376ad2f680907766e6ebd6fbb5ff9841116e7b9396907 Jul 31 15:37:26.265504 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:26.265417 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" event={"ID":"70cd2b93-b97c-41ff-81f4-b3aa89f2347c","Type":"ContainerStarted","Data":"58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822"} Jul 31 15:37:26.265504 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:26.265468 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" event={"ID":"70cd2b93-b97c-41ff-81f4-b3aa89f2347c","Type":"ContainerStarted","Data":"a8f6fdca0d516a02c59376ad2f680907766e6ebd6fbb5ff9841116e7b9396907"} Jul 31 15:37:26.265945 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:26.265520 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:37:57.272986 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:57.272953 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:37:57.291349 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:37:57.291300 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" podStartSLOduration=32.291284795 podStartE2EDuration="32.291284795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:37:26.284293486 +0000 UTC m=+2957.885729824" watchObservedRunningTime="2026-07-31 15:37:57.291284795 +0000 UTC m=+2988.892721131" Jul 31 15:38:17.254722 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:38:17.254683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:38:24.538840 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:38:24.538796 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:38:37.335991 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:38:37.335961 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:38:37.433618 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:38:37.433585 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:38:37.534337 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:38:37.534312 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:38:43.812335 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:38:43.812303 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:38:43.913641 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:38:43.913612 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:39:03.089483 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:03.089400 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:39:04.798412 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:04.798381 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:39:40.654440 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:40.654395 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:39:40.743795 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:40.743734 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh"] Jul 31 15:39:40.755281 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:40.755253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:39:44.151375 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:44.151328 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:39:45.590497 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:45.590467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:39:47.272829 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:47.272793 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" podUID="70cd2b93-b97c-41ff-81f4-b3aa89f2347c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:39:47.929420 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:47.929386 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:39:48.028311 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:48.028269 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:39:55.763163 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:55.763106 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" podUID="70cd2b93-b97c-41ff-81f4-b3aa89f2347c" containerName="main" containerID="cri-o://58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822" gracePeriod=45 Jul 31 15:39:56.018719 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.018648 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:39:56.060391 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.060359 2582 generic.go:356] "Generic (PLEG): container finished" podID="70cd2b93-b97c-41ff-81f4-b3aa89f2347c" containerID="58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822" exitCode=0 Jul 31 15:39:56.060549 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.060409 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" Jul 31 15:39:56.060549 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.060436 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" event={"ID":"70cd2b93-b97c-41ff-81f4-b3aa89f2347c","Type":"ContainerDied","Data":"58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822"} Jul 31 15:39:56.060549 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.060514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh" event={"ID":"70cd2b93-b97c-41ff-81f4-b3aa89f2347c","Type":"ContainerDied","Data":"a8f6fdca0d516a02c59376ad2f680907766e6ebd6fbb5ff9841116e7b9396907"} Jul 31 15:39:56.060549 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.060536 2582 scope.go:117] "RemoveContainer" containerID="58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822" Jul 31 15:39:56.069385 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.069366 2582 scope.go:117] "RemoveContainer" containerID="58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822" Jul 31 15:39:56.069671 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:39:56.069649 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822\": container with ID starting with 58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822 not found: ID does not exist" containerID="58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822" Jul 31 15:39:56.069752 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.069684 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822"} err="failed to get container status \"58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822\": rpc error: code = NotFound desc = could not find container \"58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822\": container with ID starting with 58d457b6de4aad8f36b7ad478f96319a6b9faf461c81c767a6b8664cdcdbc822 not found: ID does not exist" Jul 31 15:39:56.140788 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.140760 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-tls-certs\") pod \"70cd2b93-b97c-41ff-81f4-b3aa89f2347c\" (UID: \"70cd2b93-b97c-41ff-81f4-b3aa89f2347c\") " Jul 31 15:39:56.140940 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.140879 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpb4d\" (UniqueName: \"kubernetes.io/projected/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-kube-api-access-lpb4d\") pod \"70cd2b93-b97c-41ff-81f4-b3aa89f2347c\" (UID: \"70cd2b93-b97c-41ff-81f4-b3aa89f2347c\") " Jul 31 15:39:56.142950 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.142917 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70cd2b93-b97c-41ff-81f4-b3aa89f2347c" (UID: "70cd2b93-b97c-41ff-81f4-b3aa89f2347c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:39:56.143069 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.142999 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-kube-api-access-lpb4d" (OuterVolumeSpecName: "kube-api-access-lpb4d") pod "70cd2b93-b97c-41ff-81f4-b3aa89f2347c" (UID: "70cd2b93-b97c-41ff-81f4-b3aa89f2347c"). InnerVolumeSpecName "kube-api-access-lpb4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:39:56.241801 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.241769 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:39:56.241801 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.241795 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpb4d\" (UniqueName: \"kubernetes.io/projected/70cd2b93-b97c-41ff-81f4-b3aa89f2347c-kube-api-access-lpb4d\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:39:56.383380 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.383349 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh"] Jul 31 15:39:56.385495 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.385470 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-55jmh"] Jul 31 15:39:56.775840 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:39:56.775810 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70cd2b93-b97c-41ff-81f4-b3aa89f2347c" path="/var/lib/kubelet/pods/70cd2b93-b97c-41ff-81f4-b3aa89f2347c/volumes" Jul 31 15:40:00.404159 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:00.404128 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:00.499148 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:00.499098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:05.432983 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:05.432944 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:17.836920 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:17.836887 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:18.126739 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.126652 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g"] Jul 31 15:40:18.127256 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.127233 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70cd2b93-b97c-41ff-81f4-b3aa89f2347c" containerName="main" Jul 31 15:40:18.127256 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.127255 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="70cd2b93-b97c-41ff-81f4-b3aa89f2347c" containerName="main" Jul 31 15:40:18.127404 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.127350 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="70cd2b93-b97c-41ff-81f4-b3aa89f2347c" containerName="main" Jul 31 15:40:18.131746 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.131725 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:40:18.135284 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.135257 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\"" Jul 31 15:40:18.135284 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.135272 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-6d5jn\"" Jul 31 15:40:18.135478 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.135288 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\"" Jul 31 15:40:18.135478 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.135287 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\"" Jul 31 15:40:18.135478 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.135274 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-mx9jq\"" Jul 31 15:40:18.140166 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.140143 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g"] Jul 31 15:40:18.224128 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.224098 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g\" (UID: \"7ccf4c26-a620-4977-a7b9-4f2b0d55a675\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:40:18.224320 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.224141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wt5b\" (UniqueName: \"kubernetes.io/projected/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-kube-api-access-4wt5b\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g\" (UID: \"7ccf4c26-a620-4977-a7b9-4f2b0d55a675\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:40:18.299195 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.299150 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:18.325253 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.325225 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wt5b\" (UniqueName: \"kubernetes.io/projected/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-kube-api-access-4wt5b\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g\" (UID: \"7ccf4c26-a620-4977-a7b9-4f2b0d55a675\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:40:18.325367 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.325324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g\" (UID: \"7ccf4c26-a620-4977-a7b9-4f2b0d55a675\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:40:18.327785 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.327761 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g\" (UID: \"7ccf4c26-a620-4977-a7b9-4f2b0d55a675\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:40:18.333632 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.333612 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wt5b\" (UniqueName: \"kubernetes.io/projected/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-kube-api-access-4wt5b\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g\" (UID: \"7ccf4c26-a620-4977-a7b9-4f2b0d55a675\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:40:18.444210 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.444181 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:40:18.577885 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:18.577857 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g"] Jul 31 15:40:18.579423 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:40:18.579401 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccf4c26_a620_4977_a7b9_4f2b0d55a675.slice/crio-24999d4058914f447b39b2841c1090a369626626005fd8786467264fa0f12d70 WatchSource:0}: Error finding container 24999d4058914f447b39b2841c1090a369626626005fd8786467264fa0f12d70: Status 404 returned error can't find the container with id 24999d4058914f447b39b2841c1090a369626626005fd8786467264fa0f12d70 Jul 31 15:40:19.188234 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:19.188198 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" event={"ID":"7ccf4c26-a620-4977-a7b9-4f2b0d55a675","Type":"ContainerStarted","Data":"5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06"} Jul 31 15:40:19.188634 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:19.188247 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" event={"ID":"7ccf4c26-a620-4977-a7b9-4f2b0d55a675","Type":"ContainerStarted","Data":"24999d4058914f447b39b2841c1090a369626626005fd8786467264fa0f12d70"} Jul 31 15:40:19.188634 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:19.188343 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:40:19.206494 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:19.206449 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" podStartSLOduration=1.206432785 podStartE2EDuration="1.206432785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:40:19.206127219 +0000 UTC m=+3130.807563556" watchObservedRunningTime="2026-07-31 15:40:19.206432785 +0000 UTC m=+3130.807869123" Jul 31 15:40:28.660253 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:28.660223 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:29.157322 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:29.157290 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:35.233933 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:35.233901 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:35.615044 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:35.614961 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:39.480975 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:39.480927 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:39.576184 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:39.576152 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:50.196949 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:50.196921 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:40:50.312452 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:50.312425 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:50.420661 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:50.420623 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:40:50.521722 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:40:50.521644 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:41:34.972716 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:34.972684 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:41:55.026409 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:55.026375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:41:55.126339 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:55.126310 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:41:55.226812 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:55.226782 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:41:55.993804 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:55.993774 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:41:58.635493 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.635455 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:41:58.778051 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.778020 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj"] Jul 31 15:41:58.781668 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.781633 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:41:58.784469 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.784443 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj"] Jul 31 15:41:58.785357 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.785334 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\"" Jul 31 15:41:58.785357 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.785352 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\"" Jul 31 15:41:58.785640 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.785620 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-cf98k\"" Jul 31 15:41:58.785754 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.785657 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-qx2nq\"" Jul 31 15:41:58.785754 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.785683 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\"" Jul 31 15:41:58.820391 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.820364 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:41:58.919897 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.919833 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt92j\" (UniqueName: \"kubernetes.io/projected/0cf142b4-092f-4997-af18-6611bb4712f1-kube-api-access-vt92j\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-46jlj\" (UID: \"0cf142b4-092f-4997-af18-6611bb4712f1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:41:58.919897 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:58.919883 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf142b4-092f-4997-af18-6611bb4712f1-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-46jlj\" (UID: \"0cf142b4-092f-4997-af18-6611bb4712f1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:41:59.021297 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:59.021255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt92j\" (UniqueName: \"kubernetes.io/projected/0cf142b4-092f-4997-af18-6611bb4712f1-kube-api-access-vt92j\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-46jlj\" (UID: \"0cf142b4-092f-4997-af18-6611bb4712f1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:41:59.021436 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:59.021323 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf142b4-092f-4997-af18-6611bb4712f1-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-46jlj\" (UID: \"0cf142b4-092f-4997-af18-6611bb4712f1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:41:59.023797 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:59.023775 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf142b4-092f-4997-af18-6611bb4712f1-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-46jlj\" (UID: \"0cf142b4-092f-4997-af18-6611bb4712f1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:41:59.031942 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:59.031914 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt92j\" (UniqueName: \"kubernetes.io/projected/0cf142b4-092f-4997-af18-6611bb4712f1-kube-api-access-vt92j\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-46jlj\" (UID: \"0cf142b4-092f-4997-af18-6611bb4712f1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:41:59.093624 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:59.093601 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:41:59.230820 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:59.230797 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj"] Jul 31 15:41:59.232298 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:41:59.232266 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf142b4_092f_4997_af18_6611bb4712f1.slice/crio-10e03002b67e6dff61dd1a5ccd7ff6e2deb19c9506b4397607bd7d9bd1241c89 WatchSource:0}: Error finding container 10e03002b67e6dff61dd1a5ccd7ff6e2deb19c9506b4397607bd7d9bd1241c89: Status 404 returned error can't find the container with id 10e03002b67e6dff61dd1a5ccd7ff6e2deb19c9506b4397607bd7d9bd1241c89 Jul 31 15:41:59.715202 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:59.715172 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" event={"ID":"0cf142b4-092f-4997-af18-6611bb4712f1","Type":"ContainerStarted","Data":"d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e"} Jul 31 15:41:59.715596 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:59.715218 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" event={"ID":"0cf142b4-092f-4997-af18-6611bb4712f1","Type":"ContainerStarted","Data":"10e03002b67e6dff61dd1a5ccd7ff6e2deb19c9506b4397607bd7d9bd1241c89"} Jul 31 15:41:59.715596 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:59.715313 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:41:59.736362 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:41:59.736316 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" podStartSLOduration=1.736302565 podStartE2EDuration="1.736302565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:41:59.735025119 +0000 UTC m=+3231.336461456" watchObservedRunningTime="2026-07-31 15:41:59.736302565 +0000 UTC m=+3231.337738902" Jul 31 15:42:01.221391 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:01.221306 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:42:01.312808 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:01.312765 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:42:23.150781 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:23.150748 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:42:30.722618 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:30.722589 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:42:36.882495 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:36.882464 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:42:36.892180 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:36.892150 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g"] Jul 31 15:42:37.013370 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:37.013336 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:42:40.197389 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:40.197346 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" podUID="7ccf4c26-a620-4977-a7b9-4f2b0d55a675" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:42:50.196778 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:50.196736 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" podUID="7ccf4c26-a620-4977-a7b9-4f2b0d55a675" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:42:51.913343 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:51.913301 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" podUID="7ccf4c26-a620-4977-a7b9-4f2b0d55a675" containerName="main" containerID="cri-o://5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06" gracePeriod=45 Jul 31 15:42:52.170814 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:52.170758 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:42:52.258312 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:52.258285 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-tls-certs\") pod \"7ccf4c26-a620-4977-a7b9-4f2b0d55a675\" (UID: \"7ccf4c26-a620-4977-a7b9-4f2b0d55a675\") " Jul 31 15:42:52.258458 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:52.258318 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wt5b\" (UniqueName: \"kubernetes.io/projected/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-kube-api-access-4wt5b\") pod \"7ccf4c26-a620-4977-a7b9-4f2b0d55a675\" (UID: \"7ccf4c26-a620-4977-a7b9-4f2b0d55a675\") " Jul 31 15:42:52.260435 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:52.260406 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-kube-api-access-4wt5b" (OuterVolumeSpecName: "kube-api-access-4wt5b") pod "7ccf4c26-a620-4977-a7b9-4f2b0d55a675" (UID: "7ccf4c26-a620-4977-a7b9-4f2b0d55a675"). InnerVolumeSpecName "kube-api-access-4wt5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:42:52.260435 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:52.260417 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ccf4c26-a620-4977-a7b9-4f2b0d55a675" (UID: "7ccf4c26-a620-4977-a7b9-4f2b0d55a675"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:42:52.358930 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:52.358903 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:42:52.358930 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:52.358928 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wt5b\" (UniqueName: \"kubernetes.io/projected/7ccf4c26-a620-4977-a7b9-4f2b0d55a675-kube-api-access-4wt5b\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:42:53.015085 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:53.015051 2582 generic.go:356] "Generic (PLEG): container finished" podID="7ccf4c26-a620-4977-a7b9-4f2b0d55a675" containerID="5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06" exitCode=0 Jul 31 15:42:53.015466 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:53.015095 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" Jul 31 15:42:53.015466 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:53.015136 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" event={"ID":"7ccf4c26-a620-4977-a7b9-4f2b0d55a675","Type":"ContainerDied","Data":"5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06"} Jul 31 15:42:53.015466 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:53.015200 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g" event={"ID":"7ccf4c26-a620-4977-a7b9-4f2b0d55a675","Type":"ContainerDied","Data":"24999d4058914f447b39b2841c1090a369626626005fd8786467264fa0f12d70"} Jul 31 15:42:53.015466 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:53.015232 2582 scope.go:117] "RemoveContainer" containerID="5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06" Jul 31 15:42:53.025033 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:53.025008 2582 scope.go:117] "RemoveContainer" containerID="5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06" Jul 31 15:42:53.025319 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:42:53.025298 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06\": container with ID starting with 5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06 not found: ID does not exist" containerID="5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06" Jul 31 15:42:53.025384 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:53.025328 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06"} err="failed to get container status \"5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06\": rpc error: code = NotFound desc = could not find container \"5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06\": container with ID starting with 5247e2968489228f6d63b1866aecaf02290269d4b39bb20f9ea88351fd1d6f06 not found: ID does not exist" Jul 31 15:42:53.035179 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:53.035116 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g"] Jul 31 15:42:53.039079 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:53.039056 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche4qg4g"] Jul 31 15:42:54.775264 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:54.775233 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ccf4c26-a620-4977-a7b9-4f2b0d55a675" path="/var/lib/kubelet/pods/7ccf4c26-a620-4977-a7b9-4f2b0d55a675/volumes" Jul 31 15:42:57.117387 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:57.117354 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:42:57.467058 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:42:57.467005 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:43:03.634810 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:03.634775 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:43:08.224276 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.224225 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:43:08.450880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.450846 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc"] Jul 31 15:43:08.451404 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.451383 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ccf4c26-a620-4977-a7b9-4f2b0d55a675" containerName="main" Jul 31 15:43:08.451451 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.451408 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ccf4c26-a620-4977-a7b9-4f2b0d55a675" containerName="main" Jul 31 15:43:08.451504 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.451494 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ccf4c26-a620-4977-a7b9-4f2b0d55a675" containerName="main" Jul 31 15:43:08.456199 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.456181 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:43:08.459423 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.459398 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-b5x9q\"" Jul 31 15:43:08.459737 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.459413 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-bmxgl\"" Jul 31 15:43:08.460069 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.459467 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\"" Jul 31 15:43:08.460069 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.459480 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\"" Jul 31 15:43:08.460069 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.459529 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\"" Jul 31 15:43:08.462330 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.462313 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc"] Jul 31 15:43:08.475821 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.475764 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:43:08.584626 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.584598 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ccs5\" (UniqueName: \"kubernetes.io/projected/6f424d63-f965-4a6d-9972-4937211f42f4-kube-api-access-2ccs5\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc\" (UID: \"6f424d63-f965-4a6d-9972-4937211f42f4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:43:08.584730 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.584654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f424d63-f965-4a6d-9972-4937211f42f4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc\" (UID: \"6f424d63-f965-4a6d-9972-4937211f42f4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:43:08.685919 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.685892 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f424d63-f965-4a6d-9972-4937211f42f4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc\" (UID: \"6f424d63-f965-4a6d-9972-4937211f42f4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:43:08.686023 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.685992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ccs5\" (UniqueName: \"kubernetes.io/projected/6f424d63-f965-4a6d-9972-4937211f42f4-kube-api-access-2ccs5\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc\" (UID: \"6f424d63-f965-4a6d-9972-4937211f42f4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:43:08.688375 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.688349 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f424d63-f965-4a6d-9972-4937211f42f4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc\" (UID: \"6f424d63-f965-4a6d-9972-4937211f42f4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:43:08.693906 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.693886 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ccs5\" (UniqueName: \"kubernetes.io/projected/6f424d63-f965-4a6d-9972-4937211f42f4-kube-api-access-2ccs5\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc\" (UID: \"6f424d63-f965-4a6d-9972-4937211f42f4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:43:08.770579 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.770505 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-bmxgl\"" Jul 31 15:43:08.780545 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.780527 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-b5x9q\"" Jul 31 15:43:08.789572 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:08.789544 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:43:09.128617 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:09.128590 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc"] Jul 31 15:43:09.130056 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:43:09.130037 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f424d63_f965_4a6d_9972_4937211f42f4.slice/crio-036e4fc919dc4e593394d38bc1916b3984919d2f44991b6d4500abe81acf11ea WatchSource:0}: Error finding container 036e4fc919dc4e593394d38bc1916b3984919d2f44991b6d4500abe81acf11ea: Status 404 returned error can't find the container with id 036e4fc919dc4e593394d38bc1916b3984919d2f44991b6d4500abe81acf11ea Jul 31 15:43:09.957244 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:09.957208 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:43:10.015760 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:10.015733 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:43:10.112853 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:10.112828 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" event={"ID":"6f424d63-f965-4a6d-9972-4937211f42f4","Type":"ContainerStarted","Data":"0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383"} Jul 31 15:43:10.112965 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:10.112875 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" event={"ID":"6f424d63-f965-4a6d-9972-4937211f42f4","Type":"ContainerStarted","Data":"036e4fc919dc4e593394d38bc1916b3984919d2f44991b6d4500abe81acf11ea"} Jul 31 15:43:10.113013 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:10.112970 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:43:10.132883 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:10.132834 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" podStartSLOduration=2.132820898 podStartE2EDuration="2.132820898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:43:10.129379632 +0000 UTC m=+3301.730815968" watchObservedRunningTime="2026-07-31 15:43:10.132820898 +0000 UTC m=+3301.734257235" Jul 31 15:43:41.120117 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:41.120088 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:43:47.209438 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:43:47.209405 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:03.753173 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:03.753139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:03.764338 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:03.764310 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj"] Jul 31 15:44:03.850889 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:03.850859 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:10.722911 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:10.722874 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" podUID="0cf142b4-092f-4997-af18-6611bb4712f1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:44:18.783364 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:18.783303 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" podUID="0cf142b4-092f-4997-af18-6611bb4712f1" containerName="main" containerID="cri-o://d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e" gracePeriod=45 Jul 31 15:44:19.041074 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.041018 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:44:19.054919 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.054894 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt92j\" (UniqueName: \"kubernetes.io/projected/0cf142b4-092f-4997-af18-6611bb4712f1-kube-api-access-vt92j\") pod \"0cf142b4-092f-4997-af18-6611bb4712f1\" (UID: \"0cf142b4-092f-4997-af18-6611bb4712f1\") " Jul 31 15:44:19.054981 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.054932 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf142b4-092f-4997-af18-6611bb4712f1-tls-certs\") pod \"0cf142b4-092f-4997-af18-6611bb4712f1\" (UID: \"0cf142b4-092f-4997-af18-6611bb4712f1\") " Jul 31 15:44:19.056895 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.056870 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf142b4-092f-4997-af18-6611bb4712f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cf142b4-092f-4997-af18-6611bb4712f1" (UID: "0cf142b4-092f-4997-af18-6611bb4712f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:44:19.056997 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.056967 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf142b4-092f-4997-af18-6611bb4712f1-kube-api-access-vt92j" (OuterVolumeSpecName: "kube-api-access-vt92j") pod "0cf142b4-092f-4997-af18-6611bb4712f1" (UID: "0cf142b4-092f-4997-af18-6611bb4712f1"). InnerVolumeSpecName "kube-api-access-vt92j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:44:19.155983 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.155954 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt92j\" (UniqueName: \"kubernetes.io/projected/0cf142b4-092f-4997-af18-6611bb4712f1-kube-api-access-vt92j\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:44:19.155983 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.155980 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf142b4-092f-4997-af18-6611bb4712f1-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:44:19.490649 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.490617 2582 generic.go:356] "Generic (PLEG): container finished" podID="0cf142b4-092f-4997-af18-6611bb4712f1" containerID="d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e" exitCode=0 Jul 31 15:44:19.490821 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.490661 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" Jul 31 15:44:19.490821 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.490706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" event={"ID":"0cf142b4-092f-4997-af18-6611bb4712f1","Type":"ContainerDied","Data":"d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e"} Jul 31 15:44:19.490821 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.490759 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj" event={"ID":"0cf142b4-092f-4997-af18-6611bb4712f1","Type":"ContainerDied","Data":"10e03002b67e6dff61dd1a5ccd7ff6e2deb19c9506b4397607bd7d9bd1241c89"} Jul 31 15:44:19.490821 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.490781 2582 scope.go:117] "RemoveContainer" containerID="d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e" Jul 31 15:44:19.500325 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.500303 2582 scope.go:117] "RemoveContainer" containerID="d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e" Jul 31 15:44:19.500551 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:44:19.500527 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e\": container with ID starting with d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e not found: ID does not exist" containerID="d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e" Jul 31 15:44:19.500675 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.500569 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e"} err="failed to get container status \"d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e\": rpc error: code = NotFound desc = could not find container \"d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e\": container with ID starting with d39f975f4fe84bdfedf96cd222bc0181a704e3a338700ea339c90871dc8ee31e not found: ID does not exist" Jul 31 15:44:19.513413 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.513386 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj"] Jul 31 15:44:19.516203 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:19.516181 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-46jlj"] Jul 31 15:44:20.776192 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:20.776148 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf142b4-092f-4997-af18-6611bb4712f1" path="/var/lib/kubelet/pods/0cf142b4-092f-4997-af18-6611bb4712f1/volumes" Jul 31 15:44:23.783758 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:23.783724 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:23.878249 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:23.878223 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:30.109360 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:30.109330 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:31.267982 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.267944 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:31.465589 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.465548 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc"] Jul 31 15:44:31.465938 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.465926 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf142b4-092f-4997-af18-6611bb4712f1" containerName="main" Jul 31 15:44:31.465983 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.465940 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf142b4-092f-4997-af18-6611bb4712f1" containerName="main" Jul 31 15:44:31.466021 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.466015 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf142b4-092f-4997-af18-6611bb4712f1" containerName="main" Jul 31 15:44:31.469006 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.468987 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:44:31.472719 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.472697 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\"" Jul 31 15:44:31.472852 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.472700 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\"" Jul 31 15:44:31.472852 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.472751 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\"" Jul 31 15:44:31.472852 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.472787 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-m8lw2\"" Jul 31 15:44:31.472852 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.472754 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-qblhm\"" Jul 31 15:44:31.477807 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.477787 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc"] Jul 31 15:44:31.480080 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.480053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:31.543583 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.543502 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29z46\" (UniqueName: \"kubernetes.io/projected/5fb14e56-5ad1-48de-8fa6-9e960b70c452-kube-api-access-29z46\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc\" (UID: \"5fb14e56-5ad1-48de-8fa6-9e960b70c452\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:44:31.543676 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.543618 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb14e56-5ad1-48de-8fa6-9e960b70c452-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc\" (UID: \"5fb14e56-5ad1-48de-8fa6-9e960b70c452\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:44:31.644376 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.644352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb14e56-5ad1-48de-8fa6-9e960b70c452-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc\" (UID: \"5fb14e56-5ad1-48de-8fa6-9e960b70c452\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:44:31.644512 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.644385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29z46\" (UniqueName: \"kubernetes.io/projected/5fb14e56-5ad1-48de-8fa6-9e960b70c452-kube-api-access-29z46\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc\" (UID: \"5fb14e56-5ad1-48de-8fa6-9e960b70c452\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:44:31.646816 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.646795 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb14e56-5ad1-48de-8fa6-9e960b70c452-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc\" (UID: \"5fb14e56-5ad1-48de-8fa6-9e960b70c452\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:44:31.652704 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.652683 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29z46\" (UniqueName: \"kubernetes.io/projected/5fb14e56-5ad1-48de-8fa6-9e960b70c452-kube-api-access-29z46\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc\" (UID: \"5fb14e56-5ad1-48de-8fa6-9e960b70c452\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:44:31.780800 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.780777 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:44:31.916360 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:31.916332 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc"] Jul 31 15:44:31.918670 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:44:31.918634 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb14e56_5ad1_48de_8fa6_9e960b70c452.slice/crio-b7100235e4f7a5397874dbe4cf37699b9b09f6a0e40ec843b0c59745760f848d WatchSource:0}: Error finding container b7100235e4f7a5397874dbe4cf37699b9b09f6a0e40ec843b0c59745760f848d: Status 404 returned error can't find the container with id b7100235e4f7a5397874dbe4cf37699b9b09f6a0e40ec843b0c59745760f848d Jul 31 15:44:32.563234 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:32.563199 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" event={"ID":"5fb14e56-5ad1-48de-8fa6-9e960b70c452","Type":"ContainerStarted","Data":"15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d"} Jul 31 15:44:32.563677 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:32.563250 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" event={"ID":"5fb14e56-5ad1-48de-8fa6-9e960b70c452","Type":"ContainerStarted","Data":"b7100235e4f7a5397874dbe4cf37699b9b09f6a0e40ec843b0c59745760f848d"} Jul 31 15:44:32.563677 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:32.563347 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:44:32.582103 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:32.582059 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" podStartSLOduration=1.5820462530000001 podStartE2EDuration="1.582046253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:44:32.580412848 +0000 UTC m=+3384.181849186" watchObservedRunningTime="2026-07-31 15:44:32.582046253 +0000 UTC m=+3384.183482589" Jul 31 15:44:35.956856 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:35.956818 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:36.027024 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:36.026991 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:40.141410 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:40.141376 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:44:59.944000 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:44:59.943963 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:45:03.571026 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:03.570955 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:45:27.770833 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:27.770787 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:45:27.802149 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:27.802115 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc"] Jul 31 15:45:27.898180 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:27.898140 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:45:31.120021 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:31.119984 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" podUID="6f424d63-f965-4a6d-9972-4937211f42f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:45:41.119614 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:41.119577 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" podUID="6f424d63-f965-4a6d-9972-4937211f42f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:45:42.825526 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:42.825485 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" podUID="6f424d63-f965-4a6d-9972-4937211f42f4" containerName="main" containerID="cri-o://0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383" gracePeriod=45 Jul 31 15:45:43.083321 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.083270 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:45:43.142172 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.142150 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ccs5\" (UniqueName: \"kubernetes.io/projected/6f424d63-f965-4a6d-9972-4937211f42f4-kube-api-access-2ccs5\") pod \"6f424d63-f965-4a6d-9972-4937211f42f4\" (UID: \"6f424d63-f965-4a6d-9972-4937211f42f4\") " Jul 31 15:45:43.142299 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.142231 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f424d63-f965-4a6d-9972-4937211f42f4-tls-certs\") pod \"6f424d63-f965-4a6d-9972-4937211f42f4\" (UID: \"6f424d63-f965-4a6d-9972-4937211f42f4\") " Jul 31 15:45:43.144366 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.144340 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f424d63-f965-4a6d-9972-4937211f42f4-kube-api-access-2ccs5" (OuterVolumeSpecName: "kube-api-access-2ccs5") pod "6f424d63-f965-4a6d-9972-4937211f42f4" (UID: "6f424d63-f965-4a6d-9972-4937211f42f4"). InnerVolumeSpecName "kube-api-access-2ccs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:45:43.144455 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.144377 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f424d63-f965-4a6d-9972-4937211f42f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f424d63-f965-4a6d-9972-4937211f42f4" (UID: "6f424d63-f965-4a6d-9972-4937211f42f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:45:43.243802 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.243778 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f424d63-f965-4a6d-9972-4937211f42f4-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:45:43.243802 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.243799 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ccs5\" (UniqueName: \"kubernetes.io/projected/6f424d63-f965-4a6d-9972-4937211f42f4-kube-api-access-2ccs5\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:45:43.942540 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.942507 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f424d63-f965-4a6d-9972-4937211f42f4" containerID="0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383" exitCode=0 Jul 31 15:45:43.942955 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.942568 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" Jul 31 15:45:43.942955 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.942591 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" event={"ID":"6f424d63-f965-4a6d-9972-4937211f42f4","Type":"ContainerDied","Data":"0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383"} Jul 31 15:45:43.942955 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.942646 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc" event={"ID":"6f424d63-f965-4a6d-9972-4937211f42f4","Type":"ContainerDied","Data":"036e4fc919dc4e593394d38bc1916b3984919d2f44991b6d4500abe81acf11ea"} Jul 31 15:45:43.942955 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.942668 2582 scope.go:117] "RemoveContainer" containerID="0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383" Jul 31 15:45:43.952481 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.952463 2582 scope.go:117] "RemoveContainer" containerID="0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383" Jul 31 15:45:43.952746 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:45:43.952729 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383\": container with ID starting with 0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383 not found: ID does not exist" containerID="0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383" Jul 31 15:45:43.952804 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.952754 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383"} err="failed to get container status \"0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383\": rpc error: code = NotFound desc = could not find container \"0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383\": container with ID starting with 0cc7579be06e070ff8321f90ab2a0500d39c7afad7796bf5ab540370b6814383 not found: ID does not exist" Jul 31 15:45:43.964024 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.964003 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc"] Jul 31 15:45:43.968152 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:43.968131 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74vf9pc"] Jul 31 15:45:44.775216 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:44.775185 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f424d63-f965-4a6d-9972-4937211f42f4" path="/var/lib/kubelet/pods/6f424d63-f965-4a6d-9972-4937211f42f4/volumes" Jul 31 15:45:48.434569 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:48.434533 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:45:48.534617 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:48.534588 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:45:51.687754 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:51.687724 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:45:54.421421 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:45:54.421388 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:46:00.365069 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:00.365037 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:46:00.430190 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:00.430155 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:46:02.921739 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:02.921707 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:46:03.281812 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.281722 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm"] Jul 31 15:46:03.282324 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.282296 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f424d63-f965-4a6d-9972-4937211f42f4" containerName="main" Jul 31 15:46:03.282483 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.282465 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f424d63-f965-4a6d-9972-4937211f42f4" containerName="main" Jul 31 15:46:03.282626 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.282612 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f424d63-f965-4a6d-9972-4937211f42f4" containerName="main" Jul 31 15:46:03.285699 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.285680 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:46:03.289236 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.289213 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-vvnm7\"" Jul 31 15:46:03.289350 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.289213 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\"" Jul 31 15:46:03.289350 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.289287 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\"" Jul 31 15:46:03.289350 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.289342 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-klgcq\"" Jul 31 15:46:03.289614 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.289496 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\"" Jul 31 15:46:03.296015 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.295991 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm"] Jul 31 15:46:03.319499 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.319473 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:46:03.401827 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.401786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e43e75a-930b-4eac-994b-aa555f1f7c73-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm\" (UID: \"5e43e75a-930b-4eac-994b-aa555f1f7c73\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:46:03.401982 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.401919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljq5w\" (UniqueName: \"kubernetes.io/projected/5e43e75a-930b-4eac-994b-aa555f1f7c73-kube-api-access-ljq5w\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm\" (UID: \"5e43e75a-930b-4eac-994b-aa555f1f7c73\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:46:03.414309 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.414287 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:46:03.503323 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.503291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljq5w\" (UniqueName: \"kubernetes.io/projected/5e43e75a-930b-4eac-994b-aa555f1f7c73-kube-api-access-ljq5w\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm\" (UID: \"5e43e75a-930b-4eac-994b-aa555f1f7c73\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:46:03.503479 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.503345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e43e75a-930b-4eac-994b-aa555f1f7c73-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm\" (UID: \"5e43e75a-930b-4eac-994b-aa555f1f7c73\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:46:03.505747 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.505724 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e43e75a-930b-4eac-994b-aa555f1f7c73-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm\" (UID: \"5e43e75a-930b-4eac-994b-aa555f1f7c73\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:46:03.511551 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.511520 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljq5w\" (UniqueName: \"kubernetes.io/projected/5e43e75a-930b-4eac-994b-aa555f1f7c73-kube-api-access-ljq5w\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm\" (UID: \"5e43e75a-930b-4eac-994b-aa555f1f7c73\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:46:03.598690 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.598635 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:46:03.730484 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:03.730450 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm"] Jul 31 15:46:03.731925 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:46:03.731813 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e43e75a_930b_4eac_994b_aa555f1f7c73.slice/crio-51f20be74f6063d5bd257c435da15fc0905418228fa25984845b354726d0e907 WatchSource:0}: Error finding container 51f20be74f6063d5bd257c435da15fc0905418228fa25984845b354726d0e907: Status 404 returned error can't find the container with id 51f20be74f6063d5bd257c435da15fc0905418228fa25984845b354726d0e907 Jul 31 15:46:04.048500 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:04.048463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" event={"ID":"5e43e75a-930b-4eac-994b-aa555f1f7c73","Type":"ContainerStarted","Data":"34dd33fac18b67c06f51a47c59d0c34cc810c634ddbb6cb9bad4265801993af0"} Jul 31 15:46:04.048921 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:04.048524 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" event={"ID":"5e43e75a-930b-4eac-994b-aa555f1f7c73","Type":"ContainerStarted","Data":"51f20be74f6063d5bd257c435da15fc0905418228fa25984845b354726d0e907"} Jul 31 15:46:04.048921 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:04.048587 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:46:04.065133 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:04.065076 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" podStartSLOduration=1.065058509 podStartE2EDuration="1.065058509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:46:04.064969389 +0000 UTC m=+3475.666405727" watchObservedRunningTime="2026-07-31 15:46:04.065058509 +0000 UTC m=+3475.666494847" Jul 31 15:46:35.055115 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:35.055076 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:46:37.150269 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:37.150222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:46:37.206471 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:37.206441 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:46:52.800857 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:52.800825 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc"] Jul 31 15:46:52.806301 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:52.806272 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:46:52.861541 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:52.861480 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:46:53.571164 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:46:53.571120 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" podUID="5fb14e56-5ad1-48de-8fa6-9e960b70c452" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:47:03.570158 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:03.570122 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" podUID="5fb14e56-5ad1-48de-8fa6-9e960b70c452" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:47:07.820187 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:07.820110 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" podUID="5fb14e56-5ad1-48de-8fa6-9e960b70c452" containerName="main" containerID="cri-o://15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d" gracePeriod=45 Jul 31 15:47:08.068710 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.068686 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:47:08.222729 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.222691 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29z46\" (UniqueName: \"kubernetes.io/projected/5fb14e56-5ad1-48de-8fa6-9e960b70c452-kube-api-access-29z46\") pod \"5fb14e56-5ad1-48de-8fa6-9e960b70c452\" (UID: \"5fb14e56-5ad1-48de-8fa6-9e960b70c452\") " Jul 31 15:47:08.222729 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.222747 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb14e56-5ad1-48de-8fa6-9e960b70c452-tls-certs\") pod \"5fb14e56-5ad1-48de-8fa6-9e960b70c452\" (UID: \"5fb14e56-5ad1-48de-8fa6-9e960b70c452\") " Jul 31 15:47:08.224940 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.224902 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb14e56-5ad1-48de-8fa6-9e960b70c452-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fb14e56-5ad1-48de-8fa6-9e960b70c452" (UID: "5fb14e56-5ad1-48de-8fa6-9e960b70c452"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:47:08.224940 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.224919 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb14e56-5ad1-48de-8fa6-9e960b70c452-kube-api-access-29z46" (OuterVolumeSpecName: "kube-api-access-29z46") pod "5fb14e56-5ad1-48de-8fa6-9e960b70c452" (UID: "5fb14e56-5ad1-48de-8fa6-9e960b70c452"). InnerVolumeSpecName "kube-api-access-29z46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:47:08.323629 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.323596 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29z46\" (UniqueName: \"kubernetes.io/projected/5fb14e56-5ad1-48de-8fa6-9e960b70c452-kube-api-access-29z46\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:47:08.323629 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.323623 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb14e56-5ad1-48de-8fa6-9e960b70c452-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:47:08.392899 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.392868 2582 generic.go:356] "Generic (PLEG): container finished" podID="5fb14e56-5ad1-48de-8fa6-9e960b70c452" containerID="15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d" exitCode=0 Jul 31 15:47:08.393070 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.392920 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" Jul 31 15:47:08.393070 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.392957 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" event={"ID":"5fb14e56-5ad1-48de-8fa6-9e960b70c452","Type":"ContainerDied","Data":"15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d"} Jul 31 15:47:08.393070 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.393015 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc" event={"ID":"5fb14e56-5ad1-48de-8fa6-9e960b70c452","Type":"ContainerDied","Data":"b7100235e4f7a5397874dbe4cf37699b9b09f6a0e40ec843b0c59745760f848d"} Jul 31 15:47:08.393070 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.393038 2582 scope.go:117] "RemoveContainer" containerID="15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d" Jul 31 15:47:08.401903 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.401886 2582 scope.go:117] "RemoveContainer" containerID="15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d" Jul 31 15:47:08.402158 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:47:08.402135 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d\": container with ID starting with 15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d not found: ID does not exist" containerID="15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d" Jul 31 15:47:08.402222 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.402167 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d"} err="failed to get container status \"15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d\": rpc error: code = NotFound desc = could not find container \"15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d\": container with ID starting with 15d9fe2befea23a90abee9211d8fb1dfbfc8e0ddf9a2465fe18ed048a954e81d not found: ID does not exist" Jul 31 15:47:08.415129 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.415106 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc"] Jul 31 15:47:08.418837 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.418817 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-xnrbc"] Jul 31 15:47:08.777039 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:08.777014 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fb14e56-5ad1-48de-8fa6-9e960b70c452" path="/var/lib/kubelet/pods/5fb14e56-5ad1-48de-8fa6-9e960b70c452/volumes" Jul 31 15:47:55.737605 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:55.737548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:47:55.796352 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:55.796326 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:47:57.977018 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:57.976989 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:47:58.076418 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:47:58.076381 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:48:04.387215 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:04.387141 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:48:10.221520 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:10.221491 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:48:10.323273 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:10.323246 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:48:19.622193 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.622161 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7"] Jul 31 15:48:19.622569 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.622540 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb14e56-5ad1-48de-8fa6-9e960b70c452" containerName="main" Jul 31 15:48:19.622569 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.622551 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb14e56-5ad1-48de-8fa6-9e960b70c452" containerName="main" Jul 31 15:48:19.622659 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.622629 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb14e56-5ad1-48de-8fa6-9e960b70c452" containerName="main" Jul 31 15:48:19.625619 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.625602 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.629258 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.629232 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-jsbfh\"" Jul 31 15:48:19.629385 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.629307 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\"" Jul 31 15:48:19.629441 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.629313 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\"" Jul 31 15:48:19.629524 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.629506 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\"" Jul 31 15:48:19.636956 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.636916 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:48:19.699303 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.699282 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7"] Jul 31 15:48:19.781046 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.781013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.781191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.781056 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec90f431-8f34-4e90-8897-866aa72f3326-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.781191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.781092 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.781191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.781140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.781191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.781166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzz8p\" (UniqueName: \"kubernetes.io/projected/ec90f431-8f34-4e90-8897-866aa72f3326-kube-api-access-vzz8p\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.781356 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.781194 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.881820 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.881796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec90f431-8f34-4e90-8897-866aa72f3326-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.881944 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.881833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.882031 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.882016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.882075 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.882047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzz8p\" (UniqueName: \"kubernetes.io/projected/ec90f431-8f34-4e90-8897-866aa72f3326-kube-api-access-vzz8p\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.882125 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.882082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.882181 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.882170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.882312 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.882296 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.882466 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.882442 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.882527 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.882472 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.884327 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.884306 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.884451 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.884435 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec90f431-8f34-4e90-8897-866aa72f3326-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.893718 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.893678 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzz8p\" (UniqueName: \"kubernetes.io/projected/ec90f431-8f34-4e90-8897-866aa72f3326-kube-api-access-vzz8p\") pod \"rolling-upgrade-test-kserve-74b79d7f77-77qx7\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.936448 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.936428 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:19.940659 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:19.940639 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:48:20.073390 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:20.073355 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7"] Jul 31 15:48:20.075316 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:48:20.075285 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec90f431_8f34_4e90_8897_866aa72f3326.slice/crio-5681bdabf85127f9bdcd64eeb07ec6ef1ae0c33e46a28973e35094f8a4154cfa WatchSource:0}: Error finding container 5681bdabf85127f9bdcd64eeb07ec6ef1ae0c33e46a28973e35094f8a4154cfa: Status 404 returned error can't find the container with id 5681bdabf85127f9bdcd64eeb07ec6ef1ae0c33e46a28973e35094f8a4154cfa Jul 31 15:48:20.781667 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:20.781632 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" event={"ID":"ec90f431-8f34-4e90-8897-866aa72f3326","Type":"ContainerStarted","Data":"f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6"} Jul 31 15:48:20.782041 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:20.781682 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" event={"ID":"ec90f431-8f34-4e90-8897-866aa72f3326","Type":"ContainerStarted","Data":"5681bdabf85127f9bdcd64eeb07ec6ef1ae0c33e46a28973e35094f8a4154cfa"} Jul 31 15:48:20.804049 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:20.804007 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" podStartSLOduration=1.803993016 podStartE2EDuration="1.803993016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:48:20.801967382 +0000 UTC m=+3612.403403719" watchObservedRunningTime="2026-07-31 15:48:20.803993016 +0000 UTC m=+3612.405429353" Jul 31 15:48:20.936845 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:20.936818 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:29.936772 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:29.936739 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:29.950360 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:29.950340 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:30.844788 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:30.844759 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:48:31.110191 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:31.110110 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:48:42.807756 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:42.807720 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm"] Jul 31 15:48:42.812904 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:42.812877 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:48:42.911357 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:42.911322 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:48:45.055718 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:45.055684 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" podUID="5e43e75a-930b-4eac-994b-aa555f1f7c73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:48:55.055552 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:55.055514 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" podUID="5e43e75a-930b-4eac-994b-aa555f1f7c73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 31 15:48:57.828028 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:57.827965 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" podUID="5e43e75a-930b-4eac-994b-aa555f1f7c73" containerName="main" containerID="cri-o://34dd33fac18b67c06f51a47c59d0c34cc810c634ddbb6cb9bad4265801993af0" gracePeriod=45 Jul 31 15:48:57.978530 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:57.978502 2582 generic.go:356] "Generic (PLEG): container finished" podID="5e43e75a-930b-4eac-994b-aa555f1f7c73" containerID="34dd33fac18b67c06f51a47c59d0c34cc810c634ddbb6cb9bad4265801993af0" exitCode=0 Jul 31 15:48:57.978674 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:57.978591 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" event={"ID":"5e43e75a-930b-4eac-994b-aa555f1f7c73","Type":"ContainerDied","Data":"34dd33fac18b67c06f51a47c59d0c34cc810c634ddbb6cb9bad4265801993af0"} Jul 31 15:48:58.076114 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:58.076088 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:48:58.199769 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:58.199743 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e43e75a-930b-4eac-994b-aa555f1f7c73-tls-certs\") pod \"5e43e75a-930b-4eac-994b-aa555f1f7c73\" (UID: \"5e43e75a-930b-4eac-994b-aa555f1f7c73\") " Jul 31 15:48:58.199907 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:58.199804 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljq5w\" (UniqueName: \"kubernetes.io/projected/5e43e75a-930b-4eac-994b-aa555f1f7c73-kube-api-access-ljq5w\") pod \"5e43e75a-930b-4eac-994b-aa555f1f7c73\" (UID: \"5e43e75a-930b-4eac-994b-aa555f1f7c73\") " Jul 31 15:48:58.201829 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:58.201801 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e43e75a-930b-4eac-994b-aa555f1f7c73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e43e75a-930b-4eac-994b-aa555f1f7c73" (UID: "5e43e75a-930b-4eac-994b-aa555f1f7c73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:48:58.201929 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:58.201853 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e43e75a-930b-4eac-994b-aa555f1f7c73-kube-api-access-ljq5w" (OuterVolumeSpecName: "kube-api-access-ljq5w") pod "5e43e75a-930b-4eac-994b-aa555f1f7c73" (UID: "5e43e75a-930b-4eac-994b-aa555f1f7c73"). InnerVolumeSpecName "kube-api-access-ljq5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:48:58.300686 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:58.300664 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e43e75a-930b-4eac-994b-aa555f1f7c73-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:48:58.300686 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:58.300685 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljq5w\" (UniqueName: \"kubernetes.io/projected/5e43e75a-930b-4eac-994b-aa555f1f7c73-kube-api-access-ljq5w\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:48:58.985665 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:58.985633 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" Jul 31 15:48:58.986068 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:58.985674 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm" event={"ID":"5e43e75a-930b-4eac-994b-aa555f1f7c73","Type":"ContainerDied","Data":"51f20be74f6063d5bd257c435da15fc0905418228fa25984845b354726d0e907"} Jul 31 15:48:58.986068 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:58.985738 2582 scope.go:117] "RemoveContainer" containerID="34dd33fac18b67c06f51a47c59d0c34cc810c634ddbb6cb9bad4265801993af0" Jul 31 15:48:59.002199 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:59.002172 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm"] Jul 31 15:48:59.011821 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:48:59.008141 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffxqqmm"] Jul 31 15:49:00.775912 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:00.775882 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e43e75a-930b-4eac-994b-aa555f1f7c73" path="/var/lib/kubelet/pods/5e43e75a-930b-4eac-994b-aa555f1f7c73/volumes" Jul 31 15:49:02.838115 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:02.838080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:02.950610 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:02.950574 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:04.553967 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:04.553929 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7"] Jul 31 15:49:04.563966 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:49:04.563929 2582 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 31 15:49:04.563966 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:49:04.563966 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:49:04.563966 ip-10-0-135-143 kubenswrapper[2582]: > containerID="f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6" cmd=["/bin/sleep","15"] Jul 31 15:49:04.564271 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:04.563987 2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" message="" err=< Jul 31 15:49:04.564271 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:49:04.564271 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:49:04.564271 ip-10-0-135-143 kubenswrapper[2582]: > Jul 31 15:49:04.564271 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:49:04.564004 2582 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 31 15:49:04.564271 ip-10-0-135-143 kubenswrapper[2582]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 31 15:49:04.564271 ip-10-0-135-143 kubenswrapper[2582]: , stdout: , stderr: , exit code -1 Jul 31 15:49:04.564271 ip-10-0-135-143 kubenswrapper[2582]: > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" podUID="ec90f431-8f34-4e90-8897-866aa72f3326" containerName="main" containerID="cri-o://f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6" Jul 31 15:49:04.564271 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:04.564058 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" podUID="ec90f431-8f34-4e90-8897-866aa72f3326" containerName="main" containerID="cri-o://f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6" gracePeriod=60 Jul 31 15:49:04.916306 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:04.916282 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:49:05.020800 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.020768 2582 generic.go:356] "Generic (PLEG): container finished" podID="ec90f431-8f34-4e90-8897-866aa72f3326" containerID="f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6" exitCode=0 Jul 31 15:49:05.020968 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.020819 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" Jul 31 15:49:05.020968 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.020849 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" event={"ID":"ec90f431-8f34-4e90-8897-866aa72f3326","Type":"ContainerDied","Data":"f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6"} Jul 31 15:49:05.020968 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.020930 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" event={"ID":"ec90f431-8f34-4e90-8897-866aa72f3326","Type":"ContainerDied","Data":"5681bdabf85127f9bdcd64eeb07ec6ef1ae0c33e46a28973e35094f8a4154cfa"} Jul 31 15:49:05.020968 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.020953 2582 scope.go:117] "RemoveContainer" containerID="f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6" Jul 31 15:49:05.029835 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.029819 2582 scope.go:117] "RemoveContainer" containerID="f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6" Jul 31 15:49:05.030103 ip-10-0-135-143 kubenswrapper[2582]: E0731 15:49:05.030077 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6\": container with ID starting with f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6 not found: ID does not exist" containerID="f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6" Jul 31 15:49:05.030161 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.030113 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6"} err="failed to get container status \"f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6\": rpc error: code = NotFound desc = could not find container \"f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6\": container with ID starting with f57d537394f0e01a68ddc227bc6155200bd033d6842f17a3d4f5dd9052c407f6 not found: ID does not exist" Jul 31 15:49:05.060779 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.060753 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-dshm\") pod \"ec90f431-8f34-4e90-8897-866aa72f3326\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " Jul 31 15:49:05.060779 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.060782 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-tmp-dir\") pod \"ec90f431-8f34-4e90-8897-866aa72f3326\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " Jul 31 15:49:05.061006 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.060837 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-model-cache\") pod \"ec90f431-8f34-4e90-8897-866aa72f3326\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " Jul 31 15:49:05.061006 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.060855 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-home\") pod \"ec90f431-8f34-4e90-8897-866aa72f3326\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " Jul 31 15:49:05.061006 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.060876 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzz8p\" (UniqueName: \"kubernetes.io/projected/ec90f431-8f34-4e90-8897-866aa72f3326-kube-api-access-vzz8p\") pod \"ec90f431-8f34-4e90-8897-866aa72f3326\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " Jul 31 15:49:05.061006 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.060892 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec90f431-8f34-4e90-8897-866aa72f3326-tls-certs\") pod \"ec90f431-8f34-4e90-8897-866aa72f3326\" (UID: \"ec90f431-8f34-4e90-8897-866aa72f3326\") " Jul 31 15:49:05.061215 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.061090 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec90f431-8f34-4e90-8897-866aa72f3326" (UID: "ec90f431-8f34-4e90-8897-866aa72f3326"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:49:05.061215 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.061116 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec90f431-8f34-4e90-8897-866aa72f3326" (UID: "ec90f431-8f34-4e90-8897-866aa72f3326"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:49:05.061215 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.061178 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:49:05.061215 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.061183 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-home" (OuterVolumeSpecName: "home") pod "ec90f431-8f34-4e90-8897-866aa72f3326" (UID: "ec90f431-8f34-4e90-8897-866aa72f3326"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:49:05.062853 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.062831 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-dshm" (OuterVolumeSpecName: "dshm") pod "ec90f431-8f34-4e90-8897-866aa72f3326" (UID: "ec90f431-8f34-4e90-8897-866aa72f3326"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:49:05.063080 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.063060 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec90f431-8f34-4e90-8897-866aa72f3326-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec90f431-8f34-4e90-8897-866aa72f3326" (UID: "ec90f431-8f34-4e90-8897-866aa72f3326"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:49:05.063147 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.063085 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec90f431-8f34-4e90-8897-866aa72f3326-kube-api-access-vzz8p" (OuterVolumeSpecName: "kube-api-access-vzz8p") pod "ec90f431-8f34-4e90-8897-866aa72f3326" (UID: "ec90f431-8f34-4e90-8897-866aa72f3326"). InnerVolumeSpecName "kube-api-access-vzz8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:49:05.162094 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.162063 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:49:05.162094 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.162091 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:49:05.162282 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.162104 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec90f431-8f34-4e90-8897-866aa72f3326-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:49:05.162282 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.162116 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzz8p\" (UniqueName: \"kubernetes.io/projected/ec90f431-8f34-4e90-8897-866aa72f3326-kube-api-access-vzz8p\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:49:05.162282 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.162128 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec90f431-8f34-4e90-8897-866aa72f3326-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\"" Jul 31 15:49:05.345529 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.345504 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7"] Jul 31 15:49:05.349535 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.349509 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7"] Jul 31 15:49:05.834256 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:05.834208 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-77qx7" podUID="ec90f431-8f34-4e90-8897-866aa72f3326" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Jul 31 15:49:06.775513 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:06.775479 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec90f431-8f34-4e90-8897-866aa72f3326" path="/var/lib/kubelet/pods/ec90f431-8f34-4e90-8897-866aa72f3326/volumes" Jul 31 15:49:07.830531 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:07.830488 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:11.349137 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:11.349106 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:12.612234 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:12.612196 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:12.706393 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:12.706361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:28.926893 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:28.926862 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:30.915079 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:30.914980 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:30.990744 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:30.990699 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:37.108082 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:37.108047 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:37.306050 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:37.306024 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:43.048573 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:43.048529 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:43.149021 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:43.148992 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:50.160248 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:50.160220 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:52.103332 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:52.103287 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:49:52.158316 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:49:52.158268 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:50:19.226326 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:50:19.226266 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:50:36.405111 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:50:36.405062 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:51:26.487025 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:51:26.486924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:51:46.368016 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:51:46.367965 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:51:50.193843 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:51:50.193803 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:51:50.290312 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:51:50.290288 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:51:52.452382 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:51:52.452351 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:51:59.551668 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:51:59.551621 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:51:59.606072 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:51:59.606039 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:52:12.728339 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:12.728309 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-csdps_2d9b2557-cc06-4cd1-bce2-3c650fd86bc3/discovery/0.log" Jul 31 15:52:12.830709 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:12.830679 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cd69d8f8b-nwszz_277d6797-d76b-4075-9963-c88cecf9eacb/router/0.log" Jul 31 15:52:13.668986 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:13.668948 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-csdps_2d9b2557-cc06-4cd1-bce2-3c650fd86bc3/discovery/0.log" Jul 31 15:52:13.738017 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:13.737990 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cd69d8f8b-nwszz_277d6797-d76b-4075-9963-c88cecf9eacb/router/0.log" Jul 31 15:52:14.777604 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:14.777573 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-7grjx_1980debd-6ca2-42d6-943a-246d406d5b32/manager/0.log" Jul 31 15:52:14.795891 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:14.795866 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-lkd9b_044753fc-40f4-405c-8e9d-7a7d3bffa5d1/manager/0.log" Jul 31 15:52:15.029547 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:15.029446 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-zwvm4_32cf4f4f-23b5-4ad9-9e48-4da4cb9dac94/manager/0.log" Jul 31 15:52:20.932710 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:20.932672 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:52:21.072467 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.072440 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bs68h/must-gather-phmsp"] Jul 31 15:52:21.072850 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.072836 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e43e75a-930b-4eac-994b-aa555f1f7c73" containerName="main" Jul 31 15:52:21.072901 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.072852 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e43e75a-930b-4eac-994b-aa555f1f7c73" containerName="main" Jul 31 15:52:21.072901 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.072868 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec90f431-8f34-4e90-8897-866aa72f3326" containerName="main" Jul 31 15:52:21.072901 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.072874 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec90f431-8f34-4e90-8897-866aa72f3326" containerName="main" Jul 31 15:52:21.073005 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.072940 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e43e75a-930b-4eac-994b-aa555f1f7c73" containerName="main" Jul 31 15:52:21.073005 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.072950 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec90f431-8f34-4e90-8897-866aa72f3326" containerName="main" Jul 31 15:52:21.076256 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.076238 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bs68h/must-gather-phmsp" Jul 31 15:52:21.078513 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.078494 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bs68h\"/\"kube-root-ca.crt\"" Jul 31 15:52:21.078628 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.078525 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bs68h\"/\"default-dockercfg-85wq2\"" Jul 31 15:52:21.079173 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.079159 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bs68h\"/\"openshift-service-ca.crt\"" Jul 31 15:52:21.083837 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.083817 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bs68h/must-gather-phmsp"] Jul 31 15:52:21.085679 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.085653 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:52:21.157419 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.157389 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7af07fd2-7d51-41b9-8dd5-52379f529801-must-gather-output\") pod \"must-gather-phmsp\" (UID: \"7af07fd2-7d51-41b9-8dd5-52379f529801\") " pod="openshift-must-gather-bs68h/must-gather-phmsp" Jul 31 15:52:21.157579 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.157447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldpfs\" (UniqueName: \"kubernetes.io/projected/7af07fd2-7d51-41b9-8dd5-52379f529801-kube-api-access-ldpfs\") pod \"must-gather-phmsp\" (UID: \"7af07fd2-7d51-41b9-8dd5-52379f529801\") " pod="openshift-must-gather-bs68h/must-gather-phmsp" Jul 31 15:52:21.185719 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.185656 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:52:21.258082 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.258055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7af07fd2-7d51-41b9-8dd5-52379f529801-must-gather-output\") pod \"must-gather-phmsp\" (UID: \"7af07fd2-7d51-41b9-8dd5-52379f529801\") " pod="openshift-must-gather-bs68h/must-gather-phmsp" Jul 31 15:52:21.258190 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.258108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpfs\" (UniqueName: \"kubernetes.io/projected/7af07fd2-7d51-41b9-8dd5-52379f529801-kube-api-access-ldpfs\") pod \"must-gather-phmsp\" (UID: \"7af07fd2-7d51-41b9-8dd5-52379f529801\") " pod="openshift-must-gather-bs68h/must-gather-phmsp" Jul 31 15:52:21.258373 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.258357 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7af07fd2-7d51-41b9-8dd5-52379f529801-must-gather-output\") pod \"must-gather-phmsp\" (UID: \"7af07fd2-7d51-41b9-8dd5-52379f529801\") " pod="openshift-must-gather-bs68h/must-gather-phmsp" Jul 31 15:52:21.266024 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.265996 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldpfs\" (UniqueName: \"kubernetes.io/projected/7af07fd2-7d51-41b9-8dd5-52379f529801-kube-api-access-ldpfs\") pod \"must-gather-phmsp\" (UID: \"7af07fd2-7d51-41b9-8dd5-52379f529801\") " pod="openshift-must-gather-bs68h/must-gather-phmsp" Jul 31 15:52:21.394363 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.394335 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bs68h/must-gather-phmsp" Jul 31 15:52:21.521028 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.520996 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bs68h/must-gather-phmsp"] Jul 31 15:52:21.524387 ip-10-0-135-143 kubenswrapper[2582]: W0731 15:52:21.524361 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7af07fd2_7d51_41b9_8dd5_52379f529801.slice/crio-76b079050ecea3aaafab9d5dfce59466af5a23ef855ccd9e2058a5e890768439 WatchSource:0}: Error finding container 76b079050ecea3aaafab9d5dfce59466af5a23ef855ccd9e2058a5e890768439: Status 404 returned error can't find the container with id 76b079050ecea3aaafab9d5dfce59466af5a23ef855ccd9e2058a5e890768439 Jul 31 15:52:21.526238 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:21.526219 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 15:52:22.066620 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:22.066588 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bs68h/must-gather-phmsp" event={"ID":"7af07fd2-7d51-41b9-8dd5-52379f529801","Type":"ContainerStarted","Data":"76b079050ecea3aaafab9d5dfce59466af5a23ef855ccd9e2058a5e890768439"} Jul 31 15:52:23.076646 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:23.076605 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bs68h/must-gather-phmsp" event={"ID":"7af07fd2-7d51-41b9-8dd5-52379f529801","Type":"ContainerStarted","Data":"3c3422630891587fecf03086349b9a622a022e573b17b9a5cdff7c487ec3efb9"} Jul 31 15:52:23.077037 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:23.076665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bs68h/must-gather-phmsp" event={"ID":"7af07fd2-7d51-41b9-8dd5-52379f529801","Type":"ContainerStarted","Data":"afe9238ef7bf257919981e5ba823b7b4e11ae89acb81a5eb0237de57d5da43e5"} Jul 31 15:52:23.095694 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:23.095635 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bs68h/must-gather-phmsp" podStartSLOduration=1.217103921 podStartE2EDuration="2.095614501s" totalImagesPullingTime="878.51058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:52:23.092722296 +0000 UTC m=+3854.694158634" watchObservedRunningTime="2026-07-31 15:52:23.095614501 +0000 UTC m=+3854.697050840" Jul 31 15:52:24.034256 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:24.034230 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tx9n9_fa25e8f1-1145-4595-ae2a-5733ae0fc8fb/global-pull-secret-syncer/0.log" Jul 31 15:52:24.080713 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:24.080666 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" Jul 31 15:52:24.197641 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:24.197607 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-143.ec2.internal_a155768c4417dbe455678d25197f0bfd/haproxy/0.log" Jul 31 15:52:27.167778 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:27.167722 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:52:27.267888 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:27.267837 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2xz8_f56ffbbe-15a6-4eaa-93fa-2c815a6cf635/konnectivity-agent/0.log" limit=1024 Jul 31 15:52:29.166675 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:29.166646 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-7grjx_1980debd-6ca2-42d6-943a-246d406d5b32/manager/0.log" Jul 31 15:52:29.203441 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:29.203394 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-lkd9b_044753fc-40f4-405c-8e9d-7a7d3bffa5d1/manager/0.log" Jul 31 15:52:29.502402 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:29.502309 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-zwvm4_32cf4f4f-23b5-4ad9-9e48-4da4cb9dac94/manager/0.log" Jul 31 15:52:31.116383 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:31.116297 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-ks86x_6a232bc6-eeec-4edb-ae6c-2e89a68a4044/kube-state-metrics/0.log" Jul 31 15:52:31.147845 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:31.147815 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-ks86x_6a232bc6-eeec-4edb-ae6c-2e89a68a4044/kube-rbac-proxy-main/0.log" Jul 31 15:52:31.178022 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:31.177991 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-ks86x_6a232bc6-eeec-4edb-ae6c-2e89a68a4044/kube-rbac-proxy-self/0.log" Jul 31 15:52:31.221278 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:31.221228 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7584bc8c56-cnkqv_6de51fd1-134a-4066-a999-c62eb56cf797/metrics-server/0.log" Jul 31 15:52:31.310607 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:31.310578 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nvvb_dbeb1b85-9613-4cee-a362-1e8f60971b25/node-exporter/0.log" Jul 31 15:52:31.351340 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:31.351315 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nvvb_dbeb1b85-9613-4cee-a362-1e8f60971b25/kube-rbac-proxy/0.log" Jul 31 15:52:31.391573 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:31.391531 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nvvb_dbeb1b85-9613-4cee-a362-1e8f60971b25/init-textfile/0.log" Jul 31 15:52:31.668268 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:31.668187 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-9rbf8_7b3a87a3-c8fe-45af-a748-fc5a3924c643/kube-rbac-proxy-main/0.log" Jul 31 15:52:31.694863 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:31.694830 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-9rbf8_7b3a87a3-c8fe-45af-a748-fc5a3924c643/kube-rbac-proxy-self/0.log" Jul 31 15:52:31.725675 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:31.724858 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-9rbf8_7b3a87a3-c8fe-45af-a748-fc5a3924c643/openshift-state-metrics/0.log" Jul 31 15:52:32.152130 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:32.152094 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8447f8c8d6-tnvlx_a68e2a91-d93a-4869-b109-f8b569d7bf80/thanos-query/0.log" Jul 31 15:52:32.175015 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:32.174981 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8447f8c8d6-tnvlx_a68e2a91-d93a-4869-b109-f8b569d7bf80/kube-rbac-proxy-web/0.log" Jul 31 15:52:32.198587 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:32.198527 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8447f8c8d6-tnvlx_a68e2a91-d93a-4869-b109-f8b569d7bf80/kube-rbac-proxy/0.log" Jul 31 15:52:32.223999 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:32.223971 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8447f8c8d6-tnvlx_a68e2a91-d93a-4869-b109-f8b569d7bf80/prom-label-proxy/0.log" Jul 31 15:52:32.247408 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:32.247370 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8447f8c8d6-tnvlx_a68e2a91-d93a-4869-b109-f8b569d7bf80/kube-rbac-proxy-rules/0.log" Jul 31 15:52:32.272289 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:32.272257 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8447f8c8d6-tnvlx_a68e2a91-d93a-4869-b109-f8b569d7bf80/kube-rbac-proxy-metrics/0.log" Jul 31 15:52:33.889121 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:33.889085 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q"] Jul 31 15:52:33.894893 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:33.894869 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:33.907929 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:33.907450 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q"] Jul 31 15:52:33.996894 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:33.996858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2df\" (UniqueName: \"kubernetes.io/projected/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-kube-api-access-xr2df\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:33.997073 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:33.996900 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-sys\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:33.997073 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:33.997029 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-proc\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:33.997073 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:33.997059 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-podres\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:33.997200 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:33.997107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-lib-modules\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.098684 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.098645 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2df\" (UniqueName: \"kubernetes.io/projected/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-kube-api-access-xr2df\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.098880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.098693 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-sys\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.098880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.098793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-proc\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.098880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.098817 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-podres\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.098880 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.098863 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-lib-modules\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.099109 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.099019 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-lib-modules\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.099385 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.099358 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-sys\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.099457 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.099424 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-proc\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.099526 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.099510 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-podres\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.107361 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.107334 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2df\" (UniqueName: \"kubernetes.io/projected/f9cf0454-20ae-4de2-be9a-bb7def3e6fc3-kube-api-access-xr2df\") pod \"perf-node-gather-daemonset-jn88q\" (UID: \"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.210958 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.210874 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:34.297580 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.297513 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-ffd95fdd5-k42db_40296959-a9e9-4b82-a90c-dae187dcc097/console/0.log" Jul 31 15:52:34.381818 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.381777 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q"] Jul 31 15:52:34.896782 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:34.896751 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-k246r_9ff0d6ed-27d1-40aa-81da-2618aae3aa57/volume-data-source-validator/0.log" Jul 31 15:52:35.168218 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:35.168103 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" event={"ID":"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3","Type":"ContainerStarted","Data":"c0ca9198c73b038420df66d18e5ba348a3d80ddf4ad40435583d6ee1cb046183"} Jul 31 15:52:35.168378 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:35.168240 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:35.168378 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:35.168281 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" event={"ID":"f9cf0454-20ae-4de2-be9a-bb7def3e6fc3","Type":"ContainerStarted","Data":"2b35288df66de8fd72ce917b3ab7e796060453639fe10db9d9fa74e53c70bfef"} Jul 31 15:52:35.187356 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:35.187291 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" podStartSLOduration=2.187269101 podStartE2EDuration="2.187269101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 15:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:52:35.185483812 +0000 UTC m=+3866.786920149" watchObservedRunningTime="2026-07-31 15:52:35.187269101 +0000 UTC m=+3866.788705440" Jul 31 15:52:35.879290 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:35.879262 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4t89r_4f45764f-729c-43b1-8a4b-39ab982be14b/dns/0.log" Jul 31 15:52:35.901470 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:35.901448 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4t89r_4f45764f-729c-43b1-8a4b-39ab982be14b/kube-rbac-proxy/0.log" Jul 31 15:52:36.058233 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:36.058207 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tmrz2_a3b6fa83-155e-4597-a00a-6940a61cac29/dns-node-resolver/0.log" Jul 31 15:52:36.621063 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:36.621031 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-75cbcffcc4-vwhhd_2f6b0388-8dfb-4109-be06-4df54eba0704/registry/0.log" Jul 31 15:52:36.698209 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:36.698159 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s7km4_45c05bd2-6b63-4a74-86a8-f5a43da4c960/node-ca/0.log" Jul 31 15:52:37.891252 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:37.891208 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-csdps_2d9b2557-cc06-4cd1-bce2-3c650fd86bc3/discovery/0.log" Jul 31 15:52:37.971114 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:37.971073 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cd69d8f8b-nwszz_277d6797-d76b-4075-9963-c88cecf9eacb/router/0.log" Jul 31 15:52:38.527702 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:38.527673 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-js6qm_212418e4-f847-4cb7-b293-d1d119969ab3/serve-healthcheck-canary/0.log" Jul 31 15:52:39.120437 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:39.120409 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j656c_e727b6d2-1d23-42c5-bc08-42926513ede8/kube-rbac-proxy/0.log" Jul 31 15:52:39.143459 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:39.143435 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j656c_e727b6d2-1d23-42c5-bc08-42926513ede8/exporter/0.log" Jul 31 15:52:39.166239 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:39.166213 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j656c_e727b6d2-1d23-42c5-bc08-42926513ede8/extractor/0.log" Jul 31 15:52:41.189515 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:41.189484 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-jn88q" Jul 31 15:52:43.056664 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:43.056628 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-fcc9dd4db-qg2xt_4ba4ed2f-b509-43d2-8585-ea8724354232/manager/0.log" Jul 31 15:52:43.591707 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:43.591675 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-68988fb7b6-5dmnc_219152d6-7cdb-4f36-9c93-0fc8330411a4/manager/0.log" Jul 31 15:52:43.612844 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:43.612816 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-xxkbm_4def362b-7157-45ac-9a3c-0e2cf890a4af/server/0.log" Jul 31 15:52:44.131917 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:44.131861 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-csdps_2d9b2557-cc06-4cd1-bce2-3c650fd86bc3/discovery/0.log" Jul 31 15:52:44.229392 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:44.229348 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cd69d8f8b-nwszz_277d6797-d76b-4075-9963-c88cecf9eacb/router/0.log" Jul 31 15:52:44.531702 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:44.531669 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-swhnw_61904792-0b6c-462b-988a-d92f9862699b/seaweedfs/0.log" Jul 31 15:52:45.481551 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:45.481517 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-csdps_2d9b2557-cc06-4cd1-bce2-3c650fd86bc3/discovery/0.log" Jul 31 15:52:45.555781 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:45.555758 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cd69d8f8b-nwszz_277d6797-d76b-4075-9963-c88cecf9eacb/router/0.log" Jul 31 15:52:49.618388 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:49.618354 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-lsrgv_fc2efbf5-f313-46f3-895d-4185cc06f1b6/kube-storage-version-migrator-operator/1.log" Jul 31 15:52:49.619215 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:49.619199 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-lsrgv_fc2efbf5-f313-46f3-895d-4185cc06f1b6/kube-storage-version-migrator-operator/0.log" Jul 31 15:52:50.875407 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:50.875381 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84fbb_3d9ecebe-39b7-4f61-b0a3-aecbb90a9553/kube-multus-additional-cni-plugins/0.log" Jul 31 15:52:50.899113 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:50.899037 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84fbb_3d9ecebe-39b7-4f61-b0a3-aecbb90a9553/egress-router-binary-copy/0.log" Jul 31 15:52:50.920283 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:50.920250 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84fbb_3d9ecebe-39b7-4f61-b0a3-aecbb90a9553/cni-plugins/0.log" Jul 31 15:52:50.941165 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:50.941141 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84fbb_3d9ecebe-39b7-4f61-b0a3-aecbb90a9553/bond-cni-plugin/0.log" Jul 31 15:52:50.962791 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:50.962770 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84fbb_3d9ecebe-39b7-4f61-b0a3-aecbb90a9553/routeoverride-cni/0.log" Jul 31 15:52:50.984435 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:50.984415 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84fbb_3d9ecebe-39b7-4f61-b0a3-aecbb90a9553/whereabouts-cni-bincopy/0.log" Jul 31 15:52:51.010265 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:51.010239 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84fbb_3d9ecebe-39b7-4f61-b0a3-aecbb90a9553/whereabouts-cni/0.log" Jul 31 15:52:51.223600 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:51.223549 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jjklb_0f0c9dc9-912a-4ba8-9ed1-acfc78a03d28/kube-multus/0.log" Jul 31 15:52:51.377340 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:51.377310 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-thsf6_b3394f34-2e60-4e63-aee0-84ed3f6dada9/network-metrics-daemon/0.log" Jul 31 15:52:51.397528 ip-10-0-135-143 kubenswrapper[2582]: I0731 15:52:51.397492 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-thsf6_b3394f34-2e60-4e63-aee0-84ed3f6dada9/kube-rbac-proxy/0.log"