May 25 13:03:04.627512 ip-10-0-135-71 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 25 13:03:04.627669 ip-10-0-135-71 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 25 13:03:04.627754 ip-10-0-135-71 systemd[1]: kubelet.service: Failed with result 'resources'.
May 25 13:03:04.628181 ip-10-0-135-71 systemd[1]: Failed to start Kubernetes Kubelet.
May 25 13:03:14.517557 ip-10-0-135-71 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 25 13:03:14.517569 ip-10-0-135-71 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9e86edf86fdd488a975f52942e95871a --
May 25 13:05:32.606845 ip-10-0-135-71 systemd[1]: Starting Kubernetes Kubelet...
May 25 13:05:33.114195 ip-10-0-135-71 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:05:33.114195 ip-10-0-135-71 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 13:05:33.114195 ip-10-0-135-71 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:05:33.114195 ip-10-0-135-71 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 13:05:33.114195 ip-10-0-135-71 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:05:33.115296 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.115228    2573 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 13:05:33.119898 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119884    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:33.119898 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119898    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119902    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119905    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119908    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119911    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119915    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119918    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119921    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119923    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119926    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119929    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119932    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119934    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119937    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119939    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119942    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119944    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119947    2573 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119950    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119953    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:33.119963 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119956    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119959    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119961    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119964    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119967    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119970    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119977    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119982    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119986    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119989    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119992    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119995    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.119997    2573 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120000    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120003    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120006    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120009    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120012    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120015    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:33.120431 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120018    2573 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120020    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120024    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120028    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120031    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120033    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120036    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120038    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120041    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120043    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120046    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120048    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120051    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120054    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120056    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120060    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120063    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120066    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120069    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:33.120915 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120071    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120074    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120076    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120079    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120081    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120084    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120086    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120089    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120091    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120093    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120096    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120099    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120101    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120104    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120106    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120109    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120111    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120114    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120117    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120119    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:33.121374 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120122    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120124    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120127    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120130    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120132    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120135    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120137    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120476    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120481    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120484    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120487    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120490    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120492    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120495    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120497    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120500    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120503    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120505    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120508    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120510    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:33.121865 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120513    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120515    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120518    2573 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120521    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120523    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120526    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120530    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120532    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120534    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120537    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120540    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120557    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120560    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120562    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120565    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120568    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120570    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120573    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120575    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120578    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:33.122345 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120581    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120583    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120586    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120589    2573 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120592    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120594    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120597    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120599    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120603    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120606    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120608    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120611    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120613    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120616    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120618    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120621    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120623    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120626    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120629    2573 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120631    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:33.122851 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120635    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120637    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120640    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120643    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120645    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120648    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120650    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120653    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120655    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120658    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120660    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120663    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120665    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120668    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120671    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120673    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120676    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120678    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120681    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120683    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:33.123343 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120688    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120691    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120693    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120696    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120700    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120704    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120708    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120712    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120715    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120718    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120721    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120725    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.120728    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121569    2573 flags.go:64] FLAG: --address="0.0.0.0"
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121578    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121584    2573 flags.go:64] FLAG: --anonymous-auth="true"
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121589    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121594    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121597    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121601    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 13:05:33.123917 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121606    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121609    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121612    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121615    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121619    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121622    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121625    2573 flags.go:64] FLAG: --cgroup-root=""
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121628    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121631    2573 flags.go:64] FLAG: --client-ca-file=""
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121634    2573 flags.go:64] FLAG: --cloud-config=""
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121637    2573 flags.go:64] FLAG: --cloud-provider="external"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121640    2573 flags.go:64] FLAG: --cluster-dns="[]"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121645    2573 flags.go:64] FLAG: --cluster-domain=""
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121649    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121652    2573 flags.go:64] FLAG: --config-dir=""
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121655    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121658    2573 flags.go:64] FLAG: --container-log-max-files="5"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121662    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121665    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121668    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121671    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121674    2573 flags.go:64] FLAG: --contention-profiling="false"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121677    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121680    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121684    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 13:05:33.124418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121687    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121691    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121694    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121697    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121700    2573 flags.go:64] FLAG: --enable-load-reader="false"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121703    2573 flags.go:64] FLAG: --enable-server="true"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121706    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121710    2573 flags.go:64] FLAG: --event-burst="100"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121713    2573 flags.go:64] FLAG: --event-qps="50"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121716    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121719    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121722    2573 flags.go:64] FLAG: --eviction-hard=""
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121725    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121728    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121731    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121735    2573 flags.go:64] FLAG: --eviction-soft=""
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121738    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121740    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121743    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121747    2573 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121750    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121753    2573 flags.go:64] FLAG: --fail-swap-on="true"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121756    2573 flags.go:64] FLAG: --feature-gates=""
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121759    2573 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121762    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 13:05:33.125039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121766    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121769    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121772    2573 flags.go:64] FLAG: --healthz-port="10248"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121775    2573 flags.go:64] FLAG: --help="false"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121778    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-135-71.ec2.internal"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121781    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121784    2573 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121788    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121791    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121795    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121797    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121800    2573 flags.go:64] FLAG: --image-service-endpoint=""
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121803    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121806    2573 flags.go:64] FLAG: --kube-api-burst="100"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121809    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121812    2573 flags.go:64] FLAG: --kube-api-qps="50"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121814    2573 flags.go:64] FLAG: --kube-reserved=""
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121818    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121820    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121823    2573 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121826    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121829    2573 flags.go:64] FLAG: --lock-file=""
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121832    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121835    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 13:05:33.125677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121838    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121843    2573 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121846    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121849    2573 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121852    2573 flags.go:64] FLAG: --logging-format="text"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121854    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121858    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121861    2573 flags.go:64] FLAG: --manifest-url=""
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121863    2573 flags.go:64] FLAG: --manifest-url-header=""
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121868    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121871    2573 flags.go:64] FLAG: --max-open-files="1000000"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121875    2573 flags.go:64] FLAG: --max-pods="110"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121878    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121881    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121883    2573 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121886    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121890    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121893    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121896    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121903    2573 flags.go:64] FLAG: --node-status-max-images="50"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121906    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121909    2573 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121912    2573 flags.go:64] FLAG: --pod-cidr=""
May 25 13:05:33.126247 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121915    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121921    2573 flags.go:64] FLAG: --pod-manifest-path=""
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121923    2573 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121926    2573 flags.go:64] FLAG: --pods-per-core="0"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121929    2573 flags.go:64] FLAG: --port="10250"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121932    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121935    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a4d88257e36f97c9"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121938    2573 flags.go:64] FLAG: --qos-reserved=""
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121941    2573 flags.go:64] FLAG: --read-only-port="10255"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121944    2573 flags.go:64] FLAG: --register-node="true"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121947    2573 flags.go:64] FLAG: --register-schedulable="true"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121950    2573 flags.go:64] FLAG: --register-with-taints=""
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121953    2573 flags.go:64] FLAG: --registry-burst="10"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121956    2573 flags.go:64] FLAG: --registry-qps="5"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121959    2573 flags.go:64] FLAG: --reserved-cpus=""
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121962    2573 flags.go:64] FLAG: --reserved-memory=""
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121965    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121968    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121971    2573 flags.go:64] FLAG: --rotate-certificates="false"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121974    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121977    2573 flags.go:64] FLAG: --runonce="false"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121979    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.121996    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122000    2573 flags.go:64] FLAG: --seccomp-default="false"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122004    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122007    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 13:05:33.126843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122016    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122020    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122025    2573 flags.go:64] FLAG: --storage-driver-password="root"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122028    2573 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122031    2573 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122034    2573 flags.go:64] FLAG: --storage-driver-user="root"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122037    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122040    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122043    2573 flags.go:64] FLAG: --system-cgroups=""
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122045    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122051    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122054    2573 flags.go:64] FLAG: --tls-cert-file=""
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122057    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122061    2573 flags.go:64] FLAG: --tls-min-version=""
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122064    2573 flags.go:64] FLAG: --tls-private-key-file=""
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122067    2573 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122070    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122073    2573 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122076    2573 flags.go:64] FLAG: --v="2"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122084    2573 flags.go:64] FLAG: --version="false"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122088    2573 flags.go:64] FLAG: --vmodule=""
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122092    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.122096    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122191    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:33.127498 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122194    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122197    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122200    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122203    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122205    2573 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122208    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122210    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122213    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122218    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122221    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122225    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122228    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122230    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122232    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122235    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122237    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122240    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122242    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122245    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122248    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:33.128128 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122251    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122253    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122256    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122258    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122261    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122263    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122266    2573 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122268    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122271    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122274    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122276    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122278    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122281    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122283    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122286    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122289    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122291    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122293    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122296    2573 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122299    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:33.128706 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122303    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122306    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122311    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122315    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122318    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122321    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122324    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122327    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122329    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122332    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122334    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122337    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122340    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122342    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122344    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122347    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122350    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122353    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122355    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:33.129252 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122358    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122360    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122363    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122365    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122368    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122371    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122375    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122377    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122380    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122382    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122385    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122387    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122390    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122394    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122396    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122400    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122404    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122406    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122409    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122411    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:33.129788 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122414    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122416    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122419    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122422    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122424    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.122427    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.123241    2573 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.129335    2573 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.129351    2573 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129401    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129406    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129409    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129413    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129416    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129419    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129421    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:33.130356 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129424    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129426    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129429    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129431    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129434    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129437    2573 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129439    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129442    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129444    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129447    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129450    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129452    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129455    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129458    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129460    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129463    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129465    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129468    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129470    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129473    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:33.130840 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129476    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129479    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129481    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129484    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129488    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129490    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129493    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129496    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129498    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129501    2573 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129504    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129506    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129509    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129511    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129514    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129516    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129520    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129524    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129527    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129530    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:33.131392 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129533    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129536    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129539    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129542    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129560    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129564    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129566    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129569    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129571    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129574    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129576    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129579    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129582    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129585    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129588    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129590    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129593    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129597    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129599    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:33.131957 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129602    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129605    2573 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129607    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129610    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129613    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129616    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129619    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129621    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129624    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129626    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129629    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129631    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129634    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129637    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129639    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129642    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129644    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129647    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129650    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:33.132474 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129652    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.129657    2573 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129763    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129769    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129772    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129775    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129778    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129781    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129784    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129787    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129790    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129793    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129797    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129800    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129803    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129806    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:33.132983 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129808    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129811    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129814    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129816    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129819    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129821    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129824    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129826    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129829    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129831    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129834    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129837    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129841    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129844    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129847    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129850    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129853    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129855    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129858    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:33.133382 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129861    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129864    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129866    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129869    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129872    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129874    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129877    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129879    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129882    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129884    2573 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129887    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129890    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129893    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129897    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129900    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129902    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129905    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129908    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129910    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129913    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:33.133869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129915    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129917    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129920    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129923    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129925    2573 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129928    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129930    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129933    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129935    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129938    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129941    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129943    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129946    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129948    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129950    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129953    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129956    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129958    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129960    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129963    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:33.134375 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129966    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129968    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129971    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129974    2573 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129976    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129979    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129981    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129984    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129986    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129989    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129991    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129994    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:33.129996    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.130001    2573 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.130116    2573 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 13:05:33.134869 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.132246    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 13:05:33.135234 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.133312    2573 server.go:1019] "Starting client certificate rotation"
May 25 13:05:33.135234 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.133408    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 13:05:33.135234 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.133445    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 13:05:33.161972 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.161958    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 13:05:33.163985 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.163970    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 13:05:33.181726 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.181704    2573 log.go:25] "Validated CRI v1 runtime API"
May 25 13:05:33.187306 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.187290    2573 log.go:25] "Validated CRI v1 image API"
May 25 13:05:33.188696 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.188671    2573 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 13:05:33.191155 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.191136    2573 fs.go:135] Filesystem UUIDs: map[18859539-780e-4fda-91ae-cd8d6c667101:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d43361c2-fe8b-4712-88b3-fee41c28b930:/dev/nvme0n1p4]
May 25 13:05:33.191223 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.191156    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 13:05:33.194795 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.194779    2573 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 13:05:33.195307 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.195202    2573 manager.go:217] Machine: {Timestamp:2026-05-25 13:05:33.194697187 +0000 UTC m=+0.456300009 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099657 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ad724ffb77ba03eedb08c7579e1aa SystemUUID:ec2ad724-ffb7-7ba0-3eed-b08c7579e1aa BootID:9e86edf8-6fdd-488a-975f-52942e95871a Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:50:17:f8:4f:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:50:17:f8:4f:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:66:37:9f:5c:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 13:05:33.196004 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.195994    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 13:05:33.196076 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.196064    2573 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 13:05:33.197957 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.197932    2573 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 13:05:33.198092 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.197959    2573 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-71.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 13:05:33.198136 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.198098    2573 topology_manager.go:138] "Creating topology manager with none policy"
May 25 13:05:33.198136 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.198106    2573 container_manager_linux.go:306] "Creating device plugin manager"
May 25 13:05:33.198136 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.198113    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 13:05:33.199982 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.199972    2573 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 13:05:33.201790 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.201780    2573 state_mem.go:36] "Initialized new in-memory state store"
May 25 13:05:33.201893 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.201884    2573 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 13:05:33.204679 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.204669    2573 kubelet.go:491] "Attempting to sync node with API server"
May 25 13:05:33.204710 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.204689    2573 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 13:05:33.204710 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.204701    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 13:05:33.204710 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.204710    2573 kubelet.go:397] "Adding apiserver pod source"
May 25 13:05:33.204810 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.204718    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 13:05:33.205934 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.205923    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 13:05:33.205980 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.205942    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 13:05:33.209268 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.209253    2573 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 13:05:33.211717 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.211700    2573 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 13:05:33.214055 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214044    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 13:05:33.214100 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214065    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 13:05:33.214100 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214075    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 13:05:33.214100 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214083    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 13:05:33.214100 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214092    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 13:05:33.214100 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214098    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 13:05:33.214232 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214103    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 13:05:33.214232 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214130    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 13:05:33.214232 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214137    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 13:05:33.214232 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214143    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 13:05:33.214232 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214160    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 13:05:33.214232 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.214165    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 13:05:33.215152 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.215139    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 13:05:33.215210 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.215155    2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 13:05:33.216168 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.216148    2573 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 13:05:33.216485 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.216449    2573 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 13:05:33.218415 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.218401    2573 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 13:05:33.218515 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.218506    2573 server.go:1295] "Started kubelet"
May 25 13:05:33.218606 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.218562    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 13:05:33.218655 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.218601    2573 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 13:05:33.218707 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.218663    2573 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 13:05:33.219125 ip-10-0-135-71 systemd[1]: Started Kubernetes Kubelet.
May 25 13:05:33.224240 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.223653    2573 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 13:05:33.225002 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.224982    2573 server.go:317] "Adding debug handlers to kubelet server"
May 25 13:05:33.229021 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.229002    2573 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-71.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 13:05:33.230171 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.229108    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-71.ec2.internal.18b2d0c1b927a403  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-71.ec2.internal,UID:ip-10-0-135-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-71.ec2.internal,},FirstTimestamp:2026-05-25 13:05:33.218415619 +0000 UTC m=+0.480018440,LastTimestamp:2026-05-25 13:05:33.218415619 +0000 UTC m=+0.480018440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-71.ec2.internal,}"
May 25 13:05:33.230309 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.230289    2573 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 13:05:33.231410 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.231397    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 13:05:33.232619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.232042    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 13:05:33.232619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.232216    2573 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 13:05:33.232619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.232228    2573 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 13:05:33.232619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.232375    2573 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 13:05:33.232619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.232436    2573 reconstruct.go:97] "Volume reconstruction finished"
May 25 13:05:33.232619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.232447    2573 reconciler.go:26] "Reconciler: start to sync state"
May 25 13:05:33.232922 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.232820    2573 factory.go:55] Registering systemd factory
May 25 13:05:33.232922 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.232832    2573 factory.go:223] Registration of the systemd container factory successfully
May 25 13:05:33.233091 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.233072    2573 factory.go:153] Registering CRI-O factory
May 25 13:05:33.233138 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.233092    2573 factory.go:223] Registration of the crio container factory successfully
May 25 13:05:33.233294 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.233141    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 13:05:33.233376 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.233306    2573 factory.go:103] Registering Raw factory
May 25 13:05:33.233376 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.233327    2573 manager.go:1196] Started watching for new ooms in manager
May 25 13:05:33.233575 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.233557    2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:33.233734 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.233721    2573 manager.go:319] Starting recovery of all containers
May 25 13:05:33.235316 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.235288    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 13:05:33.235423 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.235314    2573 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 13:05:33.241994 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.241976    2573 manager.go:324] Recovery completed
May 25 13:05:33.242502 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.242485    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bccsb"
May 25 13:05:33.243245 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.243216    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
May 25 13:05:33.245995 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.245983    2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:33.247719 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.247699    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:33.247809 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.247733    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:33.247809 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.247743    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:33.248209 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.248195    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 13:05:33.248209 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.248208    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 13:05:33.248326 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.248225    2573 state_mem.go:36] "Initialized new in-memory state store"
May 25 13:05:33.249761 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.249703    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-71.ec2.internal.18b2d0c1bae6bc89  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-71.ec2.internal,UID:ip-10-0-135-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-71.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-71.ec2.internal,},FirstTimestamp:2026-05-25 13:05:33.247716489 +0000 UTC m=+0.509319312,LastTimestamp:2026-05-25 13:05:33.247716489 +0000 UTC m=+0.509319312,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-71.ec2.internal,}"
May 25 13:05:33.250338 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.250325    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bccsb"
May 25 13:05:33.251804 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.251791    2573 policy_none.go:49] "None policy: Start"
May 25 13:05:33.251864 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.251809    2573 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 13:05:33.251864 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.251826    2573 state_mem.go:35] "Initializing new in-memory state store"
May 25 13:05:33.286913 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.286900    2573 manager.go:341] "Starting Device Plugin manager"
May 25 13:05:33.314474 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.286928    2573 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 13:05:33.314474 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.286939    2573 server.go:85] "Starting device plugin registration server"
May 25 13:05:33.314474 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.287112    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 13:05:33.314474 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.287123    2573 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 13:05:33.314474 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.287214    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 13:05:33.314474 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.287290    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 13:05:33.314474 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.287299    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 13:05:33.314474 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.287754    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 13:05:33.314474 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.287786    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:33.331340 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.331307    2573 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 13:05:33.332465 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.332449    2573 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 13:05:33.332562 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.332474    2573 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 13:05:33.332562 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.332493    2573 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 13:05:33.332562 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.332501    2573 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 13:05:33.332562 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.332555    2573 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 13:05:33.335592 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.335572    2573 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:33.387649 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.387594    2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:33.388408 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.388393    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:33.388472 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.388416    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:33.388472 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.388426    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:33.388472 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.388445    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-71.ec2.internal"
May 25 13:05:33.399314 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.399293    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-71.ec2.internal"
May 25 13:05:33.399398 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.399316    2573 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-71.ec2.internal\": node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:33.414416 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.414397    2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:33.433410 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.433389    2573 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"]
May 25 13:05:33.433489 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.433453    2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:33.434139 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.434126    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:33.434220 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.434152    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:33.434220 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.434165    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:33.435184 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.435170    2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:33.435285 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.435272    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.435330 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.435302    2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:33.435797 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.435783    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:33.435891 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.435795    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:33.435891 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.435810    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:33.435891 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.435818    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:33.435891 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.435823    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:33.435891 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.435828    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:33.436953 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.436940    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.437016 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.436963    2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:33.437586 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.437570    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:33.437652 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.437599    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:33.437652 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.437613    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:33.461069 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.461052    2573 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-71.ec2.internal\" not found" node="ip-10-0-135-71.ec2.internal"
May 25 13:05:33.465515 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.465501    2573 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-71.ec2.internal\" not found" node="ip-10-0-135-71.ec2.internal"
May 25 13:05:33.515233 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.515213    2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:33.533747 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.533728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.533819 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.533751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.533819 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.533769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a50d4f711fbf255cb7babbc8960a87b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-71.ec2.internal\" (UID: \"9a50d4f711fbf255cb7babbc8960a87b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.615675 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.615648    2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:33.633936 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.633914    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.634017 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.633947    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a50d4f711fbf255cb7babbc8960a87b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-71.ec2.internal\" (UID: \"9a50d4f711fbf255cb7babbc8960a87b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.634017 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.633973    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.634109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.634018    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.634109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.634019    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.634109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.634021    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a50d4f711fbf255cb7babbc8960a87b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-71.ec2.internal\" (UID: \"9a50d4f711fbf255cb7babbc8960a87b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.716260 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.716219    2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:33.763598 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.763579    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.768056 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:33.768035    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 25 13:05:33.816589 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.816563    2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:33.917021 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:33.917001    2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:34.017518 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.017464    2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:34.118118 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.118084    2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 25 13:05:34.133280 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.133260    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 13:05:34.133397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.133381    2573 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 13:05:34.173338 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.173318    2573 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:34.204972 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.204944    2573 apiserver.go:52] "Watching apiserver"
May 25 13:05:34.216857 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.216832    2573 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 13:05:34.219329 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.219307    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt","openshift-multus/multus-additional-cni-plugins-mkkfs","openshift-multus/multus-rckll","openshift-ovn-kubernetes/ovnkube-node-684s7","kube-system/konnectivity-agent-c65mm","openshift-cluster-node-tuning-operator/tuned-6zsvv","openshift-image-registry/node-ca-f78nn","openshift-multus/network-metrics-daemon-tkljt","openshift-network-diagnostics/network-check-target-7xhlm","openshift-network-operator/iptables-alerter-tk4lc","kube-system/global-pull-secret-syncer-92l8w"]
May 25 13:05:34.221844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.221827    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.224275 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.224256    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 13:05:34.224275 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.224266    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 13:05:34.224410 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.224270    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 13:05:34.224410 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.224339    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k242x\""
May 25 13:05:34.225819 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.225802    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.225911 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.225894    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rckll"
May 25 13:05:34.228738 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.228717    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cdfqp\""
May 25 13:05:34.228968 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.228952    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 13:05:34.228968 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.228961    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.229084 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.229006    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 13:05:34.229084 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.229030    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 13:05:34.229172 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.229030    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 13:05:34.229371 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.229353    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 13:05:34.229445 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.229410    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ljhgm\""
May 25 13:05:34.229684 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.229671    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 13:05:34.230785 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.230765    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:34.232308 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.232270    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 13:05:34.232611 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.232567    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 13:05:34.232750 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.232738    2573 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 25 13:05:34.232902 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.232873    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 13:05:34.233299 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.233281    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.233739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.233528    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bsq9r\""
May 25 13:05:34.233739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.233636    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 13:05:34.233739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.233640    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 13:05:34.233935 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.233767    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 13:05:34.234061 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.234041    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 13:05:34.234169 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.234080    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 13:05:34.234169 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.234097    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 13:05:34.234362 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.234349    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pc95k\""
May 25 13:05:34.235823 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.235803    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.236679 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.236644    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-run-systemd\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.236781 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.236694    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 13:05:34.236781 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.236695    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-var-lib-openvswitch\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.236781 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.236739    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-cni-bin\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.236781 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.236764    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-tmp\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.236998 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.236790    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-sys-fs\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.236998 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.236879    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-hostroot\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.236998 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.236937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-conf-dir\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.236998 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.236963    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-daemon-config\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.236998 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.236987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-node-log\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.237231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237015    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-cni-netd\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.237231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237040    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 13:05:34.237231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f3d6ddb-a174-491b-b713-b4552899da35-env-overrides\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.237231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237064    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-sysconfig\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.237231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237087    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-kubernetes\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.237231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-lib-modules\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.237231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237155    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-run-ovn-kubernetes\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.237231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237187    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-cnibin\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.237231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-var-lib-kubelet\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.237231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237232    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tklm8\""
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237238    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cda7f987-514d-47c2-b386-52b0eba3d858-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-run-multus-certs\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237287    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvqwc\" (UniqueName: \"kubernetes.io/projected/b1a80a50-dca4-426d-965b-6859348c8735-kube-api-access-qvqwc\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237332    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-os-release\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237368    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-run-k8s-cni-cncf-io\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237396    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-run-openvswitch\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237424    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-os-release\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt82l\" (UniqueName: \"kubernetes.io/projected/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-kube-api-access-kt82l\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237506    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-socket-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237583    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7gzc\" (UniqueName: \"kubernetes.io/projected/cda7f987-514d-47c2-b386-52b0eba3d858-kube-api-access-p7gzc\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.237636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzk6b\" (UniqueName: \"kubernetes.io/projected/0f3d6ddb-a174-491b-b713-b4552899da35-kube-api-access-vzk6b\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237675    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-socket-dir-parent\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237696    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-run-netns\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237741    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-etc-openvswitch\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237792    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-sysctl-conf\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237821    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-system-cni-dir\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5eaa3b3-e7d0-466d-8e08-0e887a745119-cni-binary-copy\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237866    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12c6cc0f-fed3-457d-8219-eae48806d598-konnectivity-ca\") pod \"konnectivity-agent-c65mm\" (UID: \"12c6cc0f-fed3-457d-8219-eae48806d598\") " pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237889    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-host\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237904    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-system-cni-dir\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-cnibin\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-systemd\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.237983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-var-lib-cni-multus\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.237991    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpj4j\" (UniqueName: \"kubernetes.io/projected/d5eaa3b3-e7d0-466d-8e08-0e887a745119-kube-api-access-wpj4j\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238034    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-log-socket\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.238144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238057    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f3d6ddb-a174-491b-b713-b4552899da35-ovn-node-metrics-cert\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238187    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f3d6ddb-a174-491b-b713-b4552899da35-ovnkube-script-lib\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12c6cc0f-fed3-457d-8219-eae48806d598-agent-certs\") pod \"konnectivity-agent-c65mm\" (UID: \"12c6cc0f-fed3-457d-8219-eae48806d598\") " pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238242    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238272    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238314    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-registration-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238344    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-kubelet\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238367    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-systemd-units\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238370    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-run-ovn\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-sysctl-d\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238450    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-cni-dir\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238464    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9x8rn\""
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238472    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-run-netns\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238495    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-var-lib-kubelet\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238531    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-etc-kubernetes\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238581    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f3d6ddb-a174-491b-b713-b4552899da35-ovnkube-config\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.239150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238632    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-run\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.239748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-device-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.239748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cda7f987-514d-47c2-b386-52b0eba3d858-cni-binary-copy\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.239748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238718    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cda7f987-514d-47c2-b386-52b0eba3d858-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.239748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238742    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.239748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238768    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-modprobe-d\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.239748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238798    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-sys\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.239748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238828    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-tuned\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.239748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238861    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-var-lib-cni-bin\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.239748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.238887    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-slash\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.239998 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.239984    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:34.240069 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.240049    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:34.242706 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.242687    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.243133 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.243069    2573 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 13:05:34.244620 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.244603    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"]
May 25 13:05:34.244691 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.244673    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:34.244734 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.244719    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:34.245318 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.245298    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 13:05:34.245391 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.245380    2573 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 25 13:05:34.246063 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.246049    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 13:05:34.246360 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.246345    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 13:05:34.246454 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.246440    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 13:05:34.246648 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.246636    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kfwjb\""
May 25 13:05:34.252273 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.252249    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 13:00:33 +0000 UTC" deadline="2028-03-04 09:26:19.012863426 +0000 UTC"
May 25 13:05:34.252273 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.252271    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15572h20m44.76059461s"
May 25 13:05:34.264831 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.264817    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 13:05:34.265636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.265620    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"]
May 25 13:05:34.274405 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.274354    2573 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:34.320794 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:34.320768    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd208f0122d6cb3e226ed802a3b1ab4.slice/crio-66340c8798a9842c015a85479f1668bdb547910c1db3289280d0ef04d63f0642 WatchSource:0}: Error finding container 66340c8798a9842c015a85479f1668bdb547910c1db3289280d0ef04d63f0642: Status 404 returned error can't find the container with id 66340c8798a9842c015a85479f1668bdb547910c1db3289280d0ef04d63f0642
May 25 13:05:34.321043 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:34.321019    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a50d4f711fbf255cb7babbc8960a87b.slice/crio-e4ff75ed13cf9a04820af41b4ae3bc23d1ce7c32eeabeb4f87b2fde1029e6ec4 WatchSource:0}: Error finding container e4ff75ed13cf9a04820af41b4ae3bc23d1ce7c32eeabeb4f87b2fde1029e6ec4: Status 404 returned error can't find the container with id e4ff75ed13cf9a04820af41b4ae3bc23d1ce7c32eeabeb4f87b2fde1029e6ec4
May 25 13:05:34.324433 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.324407    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:05:34.333132 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.333116    2573 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 13:05:34.335113 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.335073    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal" event={"ID":"9a50d4f711fbf255cb7babbc8960a87b","Type":"ContainerStarted","Data":"e4ff75ed13cf9a04820af41b4ae3bc23d1ce7c32eeabeb4f87b2fde1029e6ec4"}
May 25 13:05:34.336220 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.336200    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal" event={"ID":"dcd208f0122d6cb3e226ed802a3b1ab4","Type":"ContainerStarted","Data":"66340c8798a9842c015a85479f1668bdb547910c1db3289280d0ef04d63f0642"}
May 25 13:05:34.339457 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339438    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-socket-dir-parent\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.339532 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339464    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-run-netns\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.339532 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339479    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-etc-openvswitch\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.339532 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339499    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-sysctl-conf\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.339532 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339521    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-system-cni-dir\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.339769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339536    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-run-netns\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.339769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339560    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5eaa3b3-e7d0-466d-8e08-0e887a745119-cni-binary-copy\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.339769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339574    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-socket-dir-parent\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.339769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339578    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-etc-openvswitch\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.339769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339588    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/880fa08b-8cfb-45ce-8c9c-b63dd8896b39-serviceca\") pod \"node-ca-f78nn\" (UID: \"880fa08b-8cfb-45ce-8c9c-b63dd8896b39\") " pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.339769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339625    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-system-cni-dir\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.339769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339678    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-sysctl-conf\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.339769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339633    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12c6cc0f-fed3-457d-8219-eae48806d598-konnectivity-ca\") pod \"konnectivity-agent-c65mm\" (UID: \"12c6cc0f-fed3-457d-8219-eae48806d598\") " pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:34.339769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339727    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-host\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.339769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339755    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-system-cni-dir\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339779    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-cnibin\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339793    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-host\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339803    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-system-cni-dir\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8269a18d-eb2f-4c7c-b747-dac1505afaba-host-slash\") pod \"iptables-alerter-tk4lc\" (UID: \"8269a18d-eb2f-4c7c-b747-dac1505afaba\") " pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430200e3-29df-46e6-99ce-346a015400ae-dbus\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339858    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-cnibin\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339867    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-systemd\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339912    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-var-lib-cni-multus\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339937    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpj4j\" (UniqueName: \"kubernetes.io/projected/d5eaa3b3-e7d0-466d-8e08-0e887a745119-kube-api-access-wpj4j\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339959    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-systemd\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339961    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-log-socket\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.339987    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-var-lib-cni-multus\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340009    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f3d6ddb-a174-491b-b713-b4552899da35-ovn-node-metrics-cert\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340020    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-log-socket\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340034    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f3d6ddb-a174-491b-b713-b4552899da35-ovnkube-script-lib\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340060    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12c6cc0f-fed3-457d-8219-eae48806d598-agent-certs\") pod \"konnectivity-agent-c65mm\" (UID: \"12c6cc0f-fed3-457d-8219-eae48806d598\") " pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:34.340173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340086    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340111    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-registration-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340126    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5eaa3b3-e7d0-466d-8e08-0e887a745119-cni-binary-copy\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340135    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-kubelet\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340159    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-systemd-units\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340205    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-systemd-units\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340112    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12c6cc0f-fed3-457d-8219-eae48806d598-konnectivity-ca\") pod \"konnectivity-agent-c65mm\" (UID: \"12c6cc0f-fed3-457d-8219-eae48806d598\") " pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340216    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-registration-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340249    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-kubelet\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340251    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-run-ovn\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340267    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340287    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-run-ovn\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340308    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340336    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-sysctl-d\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340309    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340364    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-cni-dir\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340380    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-run-netns\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340410    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-var-lib-kubelet\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.340868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340424    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-etc-kubernetes\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340441    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f3d6ddb-a174-491b-b713-b4552899da35-ovnkube-config\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340449    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-sysctl-d\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340462    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-run\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340476    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-run-netns\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340485    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-etc-kubernetes\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340493    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-device-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340480    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-var-lib-kubelet\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340511    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-cni-dir\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340519    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cda7f987-514d-47c2-b386-52b0eba3d858-cni-binary-copy\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340542    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-run\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340574    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-device-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340577    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cda7f987-514d-47c2-b386-52b0eba3d858-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340625    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340628    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f3d6ddb-a174-491b-b713-b4552899da35-ovnkube-script-lib\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340669    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340706    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-modprobe-d\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.341480 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340732    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-sys\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340754    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-tuned\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340793    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-var-lib-cni-bin\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340818    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-slash\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340824    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-sys\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340845    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpd5c\" (UniqueName: \"kubernetes.io/projected/880fa08b-8cfb-45ce-8c9c-b63dd8896b39-kube-api-access-gpd5c\") pod \"node-ca-f78nn\" (UID: \"880fa08b-8cfb-45ce-8c9c-b63dd8896b39\") " pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340872    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq44h\" (UniqueName: \"kubernetes.io/projected/c1774200-9350-464d-a3b8-2be8812bcde4-kube-api-access-vq44h\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340899    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-modprobe-d\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340906    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430200e3-29df-46e6-99ce-346a015400ae-kubelet-config\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340932    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-slash\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340932    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-run-systemd\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340942    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f3d6ddb-a174-491b-b713-b4552899da35-ovnkube-config\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340965    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-var-lib-openvswitch\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340967    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-run-systemd\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.340998    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-cni-bin\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341018    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-var-lib-cni-bin\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341029    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m6sj\" (UniqueName: \"kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj\") pod \"network-check-target-7xhlm\" (UID: \"bd26de80-992a-449c-b62d-608a45a4fa12\") " pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:34.341960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341058    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-tmp\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341061    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-var-lib-openvswitch\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341083    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cda7f987-514d-47c2-b386-52b0eba3d858-cni-binary-copy\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341097    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-sys-fs\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341099    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cda7f987-514d-47c2-b386-52b0eba3d858-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341122    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-hostroot\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341151    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-cni-bin\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341158    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-conf-dir\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341183    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-daemon-config\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341195    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-sys-fs\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341222    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-node-log\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341229    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-hostroot\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341233    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-conf-dir\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341273    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-cni-netd\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341291    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f3d6ddb-a174-491b-b713-b4552899da35-env-overrides\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341312    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-sysconfig\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341361    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-sysconfig\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341392    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-cni-netd\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.342397 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341423    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-node-log\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341464    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-kubernetes\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341530    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-lib-modules\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341585    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-run-ovn-kubernetes\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341627    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-cnibin\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341653    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-var-lib-kubelet\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341677    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cda7f987-514d-47c2-b386-52b0eba3d858-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341678    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f3d6ddb-a174-491b-b713-b4552899da35-env-overrides\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341744    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-run-multus-certs\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341752    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-lib-modules\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341785    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-run-multus-certs\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341779    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qvqwc\" (UniqueName: \"kubernetes.io/projected/b1a80a50-dca4-426d-965b-6859348c8735-kube-api-access-qvqwc\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341808    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5eaa3b3-e7d0-466d-8e08-0e887a745119-multus-daemon-config\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341821    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-os-release\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341838    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-cnibin\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341870    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-run-k8s-cni-cncf-io\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341885    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-var-lib-kubelet\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.343070 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341890    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-os-release\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341897    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-run-openvswitch\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341949    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-run-openvswitch\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.341978    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-os-release\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342006    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/880fa08b-8cfb-45ce-8c9c-b63dd8896b39-host\") pod \"node-ca-f78nn\" (UID: \"880fa08b-8cfb-45ce-8c9c-b63dd8896b39\") " pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342044    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342084    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt82l\" (UniqueName: \"kubernetes.io/projected/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-kube-api-access-kt82l\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342106    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-kubernetes\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342110    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-socket-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342168    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-host-run-k8s-cni-cncf-io\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342197    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342220    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p7gzc\" (UniqueName: \"kubernetes.io/projected/cda7f987-514d-47c2-b386-52b0eba3d858-kube-api-access-p7gzc\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342247    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8269a18d-eb2f-4c7c-b747-dac1505afaba-iptables-alerter-script\") pod \"iptables-alerter-tk4lc\" (UID: \"8269a18d-eb2f-4c7c-b747-dac1505afaba\") " pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342286    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5km7\" (UniqueName: \"kubernetes.io/projected/8269a18d-eb2f-4c7c-b747-dac1505afaba-kube-api-access-c5km7\") pod \"iptables-alerter-tk4lc\" (UID: \"8269a18d-eb2f-4c7c-b747-dac1505afaba\") " pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342314    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cda7f987-514d-47c2-b386-52b0eba3d858-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342325    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f3d6ddb-a174-491b-b713-b4552899da35-host-run-ovn-kubernetes\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342318    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.343657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342378    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vzk6b\" (UniqueName: \"kubernetes.io/projected/0f3d6ddb-a174-491b-b713-b4552899da35-kube-api-access-vzk6b\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.344207 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342488    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cda7f987-514d-47c2-b386-52b0eba3d858-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.344207 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342630    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-socket-dir\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.344207 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342669    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5eaa3b3-e7d0-466d-8e08-0e887a745119-os-release\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.344207 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.342676    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1a80a50-dca4-426d-965b-6859348c8735-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.344207 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.343831    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12c6cc0f-fed3-457d-8219-eae48806d598-agent-certs\") pod \"konnectivity-agent-c65mm\" (UID: \"12c6cc0f-fed3-457d-8219-eae48806d598\") " pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:34.344365 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.344268    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-etc-tuned\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.344365 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.344314    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f3d6ddb-a174-491b-b713-b4552899da35-ovn-node-metrics-cert\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.344365 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.344351    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-tmp\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.348800 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.348778    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpj4j\" (UniqueName: \"kubernetes.io/projected/d5eaa3b3-e7d0-466d-8e08-0e887a745119-kube-api-access-wpj4j\") pod \"multus-rckll\" (UID: \"d5eaa3b3-e7d0-466d-8e08-0e887a745119\") " pod="openshift-multus/multus-rckll"
May 25 13:05:34.350037 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.350019    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt82l\" (UniqueName: \"kubernetes.io/projected/0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb-kube-api-access-kt82l\") pod \"tuned-6zsvv\" (UID: \"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb\") " pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.350366 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.350351    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvqwc\" (UniqueName: \"kubernetes.io/projected/b1a80a50-dca4-426d-965b-6859348c8735-kube-api-access-qvqwc\") pod \"aws-ebs-csi-driver-node-6fxmt\" (UID: \"b1a80a50-dca4-426d-965b-6859348c8735\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.354026 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.354011    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7gzc\" (UniqueName: \"kubernetes.io/projected/cda7f987-514d-47c2-b386-52b0eba3d858-kube-api-access-p7gzc\") pod \"multus-additional-cni-plugins-mkkfs\" (UID: \"cda7f987-514d-47c2-b386-52b0eba3d858\") " pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.358437 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.358418    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w55bz"
May 25 13:05:34.358901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.358886    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzk6b\" (UniqueName: \"kubernetes.io/projected/0f3d6ddb-a174-491b-b713-b4552899da35-kube-api-access-vzk6b\") pod \"ovnkube-node-684s7\" (UID: \"0f3d6ddb-a174-491b-b713-b4552899da35\") " pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.365458 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.365443    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w55bz"
May 25 13:05:34.442853 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.442831    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:34.442932 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.442863    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gpd5c\" (UniqueName: \"kubernetes.io/projected/880fa08b-8cfb-45ce-8c9c-b63dd8896b39-kube-api-access-gpd5c\") pod \"node-ca-f78nn\" (UID: \"880fa08b-8cfb-45ce-8c9c-b63dd8896b39\") " pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.442932 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.442886    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vq44h\" (UniqueName: \"kubernetes.io/projected/c1774200-9350-464d-a3b8-2be8812bcde4-kube-api-access-vq44h\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:34.442932 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.442901    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430200e3-29df-46e6-99ce-346a015400ae-kubelet-config\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:34.443087 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.442974    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:05:34.443087 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443028    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430200e3-29df-46e6-99ce-346a015400ae-kubelet-config\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:34.443087 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.443038    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret podName:430200e3-29df-46e6-99ce-346a015400ae nodeName:}" failed. No retries permitted until 2026-05-25 13:05:34.943014971 +0000 UTC m=+2.204617808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret") pod "global-pull-secret-syncer-92l8w" (UID: "430200e3-29df-46e6-99ce-346a015400ae") : object "kube-system"/"original-pull-secret" not registered
May 25 13:05:34.443087 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443060    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6sj\" (UniqueName: \"kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj\") pod \"network-check-target-7xhlm\" (UID: \"bd26de80-992a-449c-b62d-608a45a4fa12\") " pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:34.443279 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443104    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/880fa08b-8cfb-45ce-8c9c-b63dd8896b39-host\") pod \"node-ca-f78nn\" (UID: \"880fa08b-8cfb-45ce-8c9c-b63dd8896b39\") " pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.443279 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443131    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:34.443279 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443158    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8269a18d-eb2f-4c7c-b747-dac1505afaba-iptables-alerter-script\") pod \"iptables-alerter-tk4lc\" (UID: \"8269a18d-eb2f-4c7c-b747-dac1505afaba\") " pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.443279 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443179    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/880fa08b-8cfb-45ce-8c9c-b63dd8896b39-host\") pod \"node-ca-f78nn\" (UID: \"880fa08b-8cfb-45ce-8c9c-b63dd8896b39\") " pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.443279 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443184    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c5km7\" (UniqueName: \"kubernetes.io/projected/8269a18d-eb2f-4c7c-b747-dac1505afaba-kube-api-access-c5km7\") pod \"iptables-alerter-tk4lc\" (UID: \"8269a18d-eb2f-4c7c-b747-dac1505afaba\") " pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.443279 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.443211    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:34.443279 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443228    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/880fa08b-8cfb-45ce-8c9c-b63dd8896b39-serviceca\") pod \"node-ca-f78nn\" (UID: \"880fa08b-8cfb-45ce-8c9c-b63dd8896b39\") " pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.443279 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.443257    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs podName:c1774200-9350-464d-a3b8-2be8812bcde4 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:34.943246459 +0000 UTC m=+2.204849284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs") pod "network-metrics-daemon-tkljt" (UID: "c1774200-9350-464d-a3b8-2be8812bcde4") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:34.443279 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443272    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8269a18d-eb2f-4c7c-b747-dac1505afaba-host-slash\") pod \"iptables-alerter-tk4lc\" (UID: \"8269a18d-eb2f-4c7c-b747-dac1505afaba\") " pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.443617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443296    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430200e3-29df-46e6-99ce-346a015400ae-dbus\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:34.443617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443326    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8269a18d-eb2f-4c7c-b747-dac1505afaba-host-slash\") pod \"iptables-alerter-tk4lc\" (UID: \"8269a18d-eb2f-4c7c-b747-dac1505afaba\") " pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.443617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443382    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430200e3-29df-46e6-99ce-346a015400ae-dbus\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:34.443617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443526    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/880fa08b-8cfb-45ce-8c9c-b63dd8896b39-serviceca\") pod \"node-ca-f78nn\" (UID: \"880fa08b-8cfb-45ce-8c9c-b63dd8896b39\") " pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.443617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.443602    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8269a18d-eb2f-4c7c-b747-dac1505afaba-iptables-alerter-script\") pod \"iptables-alerter-tk4lc\" (UID: \"8269a18d-eb2f-4c7c-b747-dac1505afaba\") " pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.451674 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.451658    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:34.451722 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.451680    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:34.451722 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.451693    2573 projected.go:194] Error preparing data for projected volume kube-api-access-4m6sj for pod openshift-network-diagnostics/network-check-target-7xhlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:34.451787 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.451756    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj podName:bd26de80-992a-449c-b62d-608a45a4fa12 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:34.951741438 +0000 UTC m=+2.213344247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4m6sj" (UniqueName: "kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj") pod "network-check-target-7xhlm" (UID: "bd26de80-992a-449c-b62d-608a45a4fa12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:34.453696 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.453675    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq44h\" (UniqueName: \"kubernetes.io/projected/c1774200-9350-464d-a3b8-2be8812bcde4-kube-api-access-vq44h\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:34.453779 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.453762    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpd5c\" (UniqueName: \"kubernetes.io/projected/880fa08b-8cfb-45ce-8c9c-b63dd8896b39-kube-api-access-gpd5c\") pod \"node-ca-f78nn\" (UID: \"880fa08b-8cfb-45ce-8c9c-b63dd8896b39\") " pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.454011 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.453997    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5km7\" (UniqueName: \"kubernetes.io/projected/8269a18d-eb2f-4c7c-b747-dac1505afaba-kube-api-access-c5km7\") pod \"iptables-alerter-tk4lc\" (UID: \"8269a18d-eb2f-4c7c-b747-dac1505afaba\") " pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.542361 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.542313    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt"
May 25 13:05:34.547841 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:34.547815    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a80a50_dca4_426d_965b_6859348c8735.slice/crio-7147e6a3d3150a5450abd093e20d85b1f12f2000581b0cf6f197e0c0279b95c3 WatchSource:0}: Error finding container 7147e6a3d3150a5450abd093e20d85b1f12f2000581b0cf6f197e0c0279b95c3: Status 404 returned error can't find the container with id 7147e6a3d3150a5450abd093e20d85b1f12f2000581b0cf6f197e0c0279b95c3
May 25 13:05:34.554453 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.554436    2573 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:34.557079 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.557063    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mkkfs"
May 25 13:05:34.562410 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:34.562392    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda7f987_514d_47c2_b386_52b0eba3d858.slice/crio-ccf4e6f287fb553b828afff5a5de1bd4b3ca571f924ed2926da5076f9e9e9052 WatchSource:0}: Error finding container ccf4e6f287fb553b828afff5a5de1bd4b3ca571f924ed2926da5076f9e9e9052: Status 404 returned error can't find the container with id ccf4e6f287fb553b828afff5a5de1bd4b3ca571f924ed2926da5076f9e9e9052
May 25 13:05:34.563143 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.563129    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rckll"
May 25 13:05:34.569199 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:34.569182    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5eaa3b3_e7d0_466d_8e08_0e887a745119.slice/crio-a60bdddb1c849c9d2f3131678ea997a93a142301ec119c29e2961b040ca031fc WatchSource:0}: Error finding container a60bdddb1c849c9d2f3131678ea997a93a142301ec119c29e2961b040ca031fc: Status 404 returned error can't find the container with id a60bdddb1c849c9d2f3131678ea997a93a142301ec119c29e2961b040ca031fc
May 25 13:05:34.585804 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.585788    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:05:34.590798 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:34.590774    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3d6ddb_a174_491b_b713_b4552899da35.slice/crio-ee0ce9b20d58c063fa2ead603d0129911e02161afa8966038086c1d965cc3773 WatchSource:0}: Error finding container ee0ce9b20d58c063fa2ead603d0129911e02161afa8966038086c1d965cc3773: Status 404 returned error can't find the container with id ee0ce9b20d58c063fa2ead603d0129911e02161afa8966038086c1d965cc3773
May 25 13:05:34.606084 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.606069    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:34.614861 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:34.614840    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c6cc0f_fed3_457d_8219_eae48806d598.slice/crio-fedcba005abbf3c2961524cd8de0a25bae502fd6b35600c78fd579c888d5d57a WatchSource:0}: Error finding container fedcba005abbf3c2961524cd8de0a25bae502fd6b35600c78fd579c888d5d57a: Status 404 returned error can't find the container with id fedcba005abbf3c2961524cd8de0a25bae502fd6b35600c78fd579c888d5d57a
May 25 13:05:34.626738 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.626722    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6zsvv"
May 25 13:05:34.631503 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:34.631479    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd7df4c_9e76_4e26_9e43_68f9a7fa05fb.slice/crio-d842c81fd1befe47f7f9d385fa40e42d9feb1ccdd67d969e7659b9b22a7b5a07 WatchSource:0}: Error finding container d842c81fd1befe47f7f9d385fa40e42d9feb1ccdd67d969e7659b9b22a7b5a07: Status 404 returned error can't find the container with id d842c81fd1befe47f7f9d385fa40e42d9feb1ccdd67d969e7659b9b22a7b5a07
May 25 13:05:34.631997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.631983    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f78nn"
May 25 13:05:34.637395 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:34.637377    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880fa08b_8cfb_45ce_8c9c_b63dd8896b39.slice/crio-21b9c6f0279a404be65fd698b5c79be974cb73a0ec8a6683cbdedc20b32e7413 WatchSource:0}: Error finding container 21b9c6f0279a404be65fd698b5c79be974cb73a0ec8a6683cbdedc20b32e7413: Status 404 returned error can't find the container with id 21b9c6f0279a404be65fd698b5c79be974cb73a0ec8a6683cbdedc20b32e7413
May 25 13:05:34.637756 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.637744    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tk4lc"
May 25 13:05:34.643166 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:05:34.643143    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8269a18d_eb2f_4c7c_b747_dac1505afaba.slice/crio-540cc81faee4c84d19ff7307eac29eb04f44aad7530402e32c2fda56cd838838 WatchSource:0}: Error finding container 540cc81faee4c84d19ff7307eac29eb04f44aad7530402e32c2fda56cd838838: Status 404 returned error can't find the container with id 540cc81faee4c84d19ff7307eac29eb04f44aad7530402e32c2fda56cd838838
May 25 13:05:34.945679 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.945605    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:34.945679 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:34.945659    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:34.945862 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.945745    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:34.945862 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.945775    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:05:34.945862 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.945811    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs podName:c1774200-9350-464d-a3b8-2be8812bcde4 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:35.945792254 +0000 UTC m=+3.207395065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs") pod "network-metrics-daemon-tkljt" (UID: "c1774200-9350-464d-a3b8-2be8812bcde4") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:34.945862 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:34.945826    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret podName:430200e3-29df-46e6-99ce-346a015400ae nodeName:}" failed. No retries permitted until 2026-05-25 13:05:35.945819865 +0000 UTC m=+3.207422674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret") pod "global-pull-secret-syncer-92l8w" (UID: "430200e3-29df-46e6-99ce-346a015400ae") : object "kube-system"/"original-pull-secret" not registered
May 25 13:05:35.046587 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.046560    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6sj\" (UniqueName: \"kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj\") pod \"network-check-target-7xhlm\" (UID: \"bd26de80-992a-449c-b62d-608a45a4fa12\") " pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:35.046730 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:35.046715    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:35.046794 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:35.046734    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:35.046794 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:35.046745    2573 projected.go:194] Error preparing data for projected volume kube-api-access-4m6sj for pod openshift-network-diagnostics/network-check-target-7xhlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:35.046794 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:35.046793    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj podName:bd26de80-992a-449c-b62d-608a45a4fa12 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:36.046777568 +0000 UTC m=+3.308380380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m6sj" (UniqueName: "kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj") pod "network-check-target-7xhlm" (UID: "bd26de80-992a-449c-b62d-608a45a4fa12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:35.335500 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.335419    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:35.336072 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:35.335610    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:35.357463 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.357427    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rckll" event={"ID":"d5eaa3b3-e7d0-466d-8e08-0e887a745119","Type":"ContainerStarted","Data":"a60bdddb1c849c9d2f3131678ea997a93a142301ec119c29e2961b040ca031fc"}
May 25 13:05:35.365294 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.365266    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tk4lc" event={"ID":"8269a18d-eb2f-4c7c-b747-dac1505afaba","Type":"ContainerStarted","Data":"540cc81faee4c84d19ff7307eac29eb04f44aad7530402e32c2fda56cd838838"}
May 25 13:05:35.366467 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.366435    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 13:00:34 +0000 UTC" deadline="2027-12-18 06:46:19.764611022 +0000 UTC"
May 25 13:05:35.366467 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.366462    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13721h40m44.3981523s"
May 25 13:05:35.374777 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.374751    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mkkfs" event={"ID":"cda7f987-514d-47c2-b386-52b0eba3d858","Type":"ContainerStarted","Data":"ccf4e6f287fb553b828afff5a5de1bd4b3ca571f924ed2926da5076f9e9e9052"}
May 25 13:05:35.383781 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.383757    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt" event={"ID":"b1a80a50-dca4-426d-965b-6859348c8735","Type":"ContainerStarted","Data":"7147e6a3d3150a5450abd093e20d85b1f12f2000581b0cf6f197e0c0279b95c3"}
May 25 13:05:35.390234 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.390203    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f78nn" event={"ID":"880fa08b-8cfb-45ce-8c9c-b63dd8896b39","Type":"ContainerStarted","Data":"21b9c6f0279a404be65fd698b5c79be974cb73a0ec8a6683cbdedc20b32e7413"}
May 25 13:05:35.399239 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.399213    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6zsvv" event={"ID":"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb","Type":"ContainerStarted","Data":"d842c81fd1befe47f7f9d385fa40e42d9feb1ccdd67d969e7659b9b22a7b5a07"}
May 25 13:05:35.401290 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.401265    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c65mm" event={"ID":"12c6cc0f-fed3-457d-8219-eae48806d598","Type":"ContainerStarted","Data":"fedcba005abbf3c2961524cd8de0a25bae502fd6b35600c78fd579c888d5d57a"}
May 25 13:05:35.420309 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.420284    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-684s7" event={"ID":"0f3d6ddb-a174-491b-b713-b4552899da35","Type":"ContainerStarted","Data":"ee0ce9b20d58c063fa2ead603d0129911e02161afa8966038086c1d965cc3773"}
May 25 13:05:35.659942 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.659494    2573 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:35.952314 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.952244    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:35.952314 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:35.952308    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:35.952523 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:35.952466    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:05:35.952828 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:35.952524    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret podName:430200e3-29df-46e6-99ce-346a015400ae nodeName:}" failed. No retries permitted until 2026-05-25 13:05:37.952504809 +0000 UTC m=+5.214107632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret") pod "global-pull-secret-syncer-92l8w" (UID: "430200e3-29df-46e6-99ce-346a015400ae") : object "kube-system"/"original-pull-secret" not registered
May 25 13:05:35.952828 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:35.952635    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:35.952828 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:35.952668    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs podName:c1774200-9350-464d-a3b8-2be8812bcde4 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:37.952657555 +0000 UTC m=+5.214260377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs") pod "network-metrics-daemon-tkljt" (UID: "c1774200-9350-464d-a3b8-2be8812bcde4") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:36.054786 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:36.054751    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6sj\" (UniqueName: \"kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj\") pod \"network-check-target-7xhlm\" (UID: \"bd26de80-992a-449c-b62d-608a45a4fa12\") " pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:36.054962 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:36.054945    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:36.055032 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:36.054968    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:36.055032 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:36.054981    2573 projected.go:194] Error preparing data for projected volume kube-api-access-4m6sj for pod openshift-network-diagnostics/network-check-target-7xhlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:36.055132 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:36.055033    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj podName:bd26de80-992a-449c-b62d-608a45a4fa12 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:38.055015366 +0000 UTC m=+5.316618193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m6sj" (UniqueName: "kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj") pod "network-check-target-7xhlm" (UID: "bd26de80-992a-449c-b62d-608a45a4fa12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:36.332786 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:36.332716    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:36.332941 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:36.332843    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:36.333091 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:36.333072    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:36.333209 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:36.333188    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:36.367493 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:36.367409    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 13:00:34 +0000 UTC" deadline="2028-03-05 22:40:51.058885864 +0000 UTC"
May 25 13:05:36.367493 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:36.367497    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15609h35m14.691398099s"
May 25 13:05:37.333541 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:37.333509    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:37.333722 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:37.333662    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:37.971313 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:37.970662    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:37.971313 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:37.970728    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:37.971313 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:37.970850    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:05:37.971313 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:37.970909    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret podName:430200e3-29df-46e6-99ce-346a015400ae nodeName:}" failed. No retries permitted until 2026-05-25 13:05:41.970891886 +0000 UTC m=+9.232494710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret") pod "global-pull-secret-syncer-92l8w" (UID: "430200e3-29df-46e6-99ce-346a015400ae") : object "kube-system"/"original-pull-secret" not registered
May 25 13:05:37.971313 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:37.971275    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:37.972063 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:37.971341    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs podName:c1774200-9350-464d-a3b8-2be8812bcde4 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:41.971321954 +0000 UTC m=+9.232924779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs") pod "network-metrics-daemon-tkljt" (UID: "c1774200-9350-464d-a3b8-2be8812bcde4") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:38.072203 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:38.071632    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6sj\" (UniqueName: \"kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj\") pod \"network-check-target-7xhlm\" (UID: \"bd26de80-992a-449c-b62d-608a45a4fa12\") " pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:38.072203 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:38.071784    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:38.072203 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:38.071802    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:38.072203 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:38.071813    2573 projected.go:194] Error preparing data for projected volume kube-api-access-4m6sj for pod openshift-network-diagnostics/network-check-target-7xhlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:38.072203 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:38.071874    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj podName:bd26de80-992a-449c-b62d-608a45a4fa12 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:42.071856529 +0000 UTC m=+9.333459348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m6sj" (UniqueName: "kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj") pod "network-check-target-7xhlm" (UID: "bd26de80-992a-449c-b62d-608a45a4fa12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:38.332985 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:38.332894    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:38.333169 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:38.333022    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:38.333243 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:38.333222    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:38.333470 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:38.333338    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:39.333099 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:39.333063    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:39.333535 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:39.333189    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:40.333068 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:40.332976    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:40.333068 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:40.332997    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:40.333582 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:40.333102    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:40.333582 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:40.333497    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:41.333340 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:41.333208    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:41.333340 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:41.333323    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:41.997448 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:41.997415    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:41.997685 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:41.997492    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:41.997685 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:41.997578    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:05:41.997685 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:41.997610    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:41.997685 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:41.997648    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret podName:430200e3-29df-46e6-99ce-346a015400ae nodeName:}" failed. No retries permitted until 2026-05-25 13:05:49.997627671 +0000 UTC m=+17.259230495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret") pod "global-pull-secret-syncer-92l8w" (UID: "430200e3-29df-46e6-99ce-346a015400ae") : object "kube-system"/"original-pull-secret" not registered
May 25 13:05:41.997685 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:41.997679    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs podName:c1774200-9350-464d-a3b8-2be8812bcde4 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:49.997668399 +0000 UTC m=+17.259271229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs") pod "network-metrics-daemon-tkljt" (UID: "c1774200-9350-464d-a3b8-2be8812bcde4") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:42.098143 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:42.098098    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6sj\" (UniqueName: \"kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj\") pod \"network-check-target-7xhlm\" (UID: \"bd26de80-992a-449c-b62d-608a45a4fa12\") " pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:42.098326 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:42.098227    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:42.098326 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:42.098251    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:42.098326 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:42.098262    2573 projected.go:194] Error preparing data for projected volume kube-api-access-4m6sj for pod openshift-network-diagnostics/network-check-target-7xhlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:42.098326 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:42.098320    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj podName:bd26de80-992a-449c-b62d-608a45a4fa12 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:50.0983022 +0000 UTC m=+17.359905027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m6sj" (UniqueName: "kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj") pod "network-check-target-7xhlm" (UID: "bd26de80-992a-449c-b62d-608a45a4fa12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:42.333148 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:42.333078    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:42.333289 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:42.333077    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:42.333289 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:42.333204    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:42.333414 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:42.333290    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:43.333757 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:43.333677    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:43.334306 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:43.333803    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:44.333555 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:44.333513    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:44.333717 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:44.333513    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:44.333717 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:44.333644    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:44.334111 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:44.333758    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:45.333078 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:45.333042    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:45.333233 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:45.333194    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:46.333144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:46.333111    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:46.333623 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:46.333122    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:46.333623 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:46.333222    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:46.333623 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:46.333341    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:47.333700 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:47.333665    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:47.334083 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:47.333789    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:48.332662 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:48.332628    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:48.332818 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:48.332633    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:48.332818 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:48.332749    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:48.332927 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:48.332831    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:49.333536 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:49.333504    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:49.333945 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:49.333650    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:50.063598 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:50.063567    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:50.063799 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:50.063625    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:50.063799 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:50.063721    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:50.063799 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:50.063724    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:05:50.063799 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:50.063772    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs podName:c1774200-9350-464d-a3b8-2be8812bcde4 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:06.063757974 +0000 UTC m=+33.325360782 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs") pod "network-metrics-daemon-tkljt" (UID: "c1774200-9350-464d-a3b8-2be8812bcde4") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:50.063799 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:50.063791    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret podName:430200e3-29df-46e6-99ce-346a015400ae nodeName:}" failed. No retries permitted until 2026-05-25 13:06:06.063780502 +0000 UTC m=+33.325383311 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret") pod "global-pull-secret-syncer-92l8w" (UID: "430200e3-29df-46e6-99ce-346a015400ae") : object "kube-system"/"original-pull-secret" not registered
May 25 13:05:50.163978 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:50.163943    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6sj\" (UniqueName: \"kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj\") pod \"network-check-target-7xhlm\" (UID: \"bd26de80-992a-449c-b62d-608a45a4fa12\") " pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:50.164132 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:50.164066    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:50.164132 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:50.164087    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:50.164132 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:50.164100    2573 projected.go:194] Error preparing data for projected volume kube-api-access-4m6sj for pod openshift-network-diagnostics/network-check-target-7xhlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:50.164260 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:50.164156    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj podName:bd26de80-992a-449c-b62d-608a45a4fa12 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:06.164139941 +0000 UTC m=+33.425742751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m6sj" (UniqueName: "kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj") pod "network-check-target-7xhlm" (UID: "bd26de80-992a-449c-b62d-608a45a4fa12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:50.332825 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:50.332752    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:50.332971 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:50.332756    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:50.332971 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:50.332879    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:50.333072 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:50.332972    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:51.332817 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:51.332790    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:51.333215 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:51.332908    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:52.333271 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:52.333241    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:52.333728 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:52.333244    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:52.333728 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:52.333372    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:52.333728 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:52.333470    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:53.333456 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:53.333434    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:53.333731 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:53.333510    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:53.457071 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:53.457038    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal" event={"ID":"9a50d4f711fbf255cb7babbc8960a87b","Type":"ContainerStarted","Data":"8c2a50b4454342e68e89bc5ed838a8431527708684d92e000e4dbe841e525c55"}
May 25 13:05:53.467838 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:53.467790    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal" podStartSLOduration=19.467774291 podStartE2EDuration="19.467774291s" podCreationTimestamp="2026-05-25 13:05:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:05:53.467071892 +0000 UTC m=+20.728674727" watchObservedRunningTime="2026-05-25 13:05:53.467774291 +0000 UTC m=+20.729377123"
May 25 13:05:54.333672 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.333403    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:54.334193 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.333403    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:54.334193 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:54.333758    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:54.334193 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:54.333833    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:54.459179 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.459148    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c65mm" event={"ID":"12c6cc0f-fed3-457d-8219-eae48806d598","Type":"ContainerStarted","Data":"42929efef91ef93f6265a28a1fded89d1f63e6a325a03448b3a2c8c174321c55"}
May 25 13:05:54.463866 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.463839    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-684s7" event={"ID":"0f3d6ddb-a174-491b-b713-b4552899da35","Type":"ContainerStarted","Data":"f6d6f88787123543f69329bc9a5c6ccb375c09926d1c8eb84c999fc8f65c8ef6"}
May 25 13:05:54.463951 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.463874    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-684s7" event={"ID":"0f3d6ddb-a174-491b-b713-b4552899da35","Type":"ContainerStarted","Data":"9283e5e2dbf6075c702d00a1b066e0e05fb4ed339652b5c25829de5cd24330c8"}
May 25 13:05:54.463951 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.463887    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-684s7" event={"ID":"0f3d6ddb-a174-491b-b713-b4552899da35","Type":"ContainerStarted","Data":"15ae3315698d5d8aeb31ec0849c45317fa6a1781c7403f56283f5e5b93158e26"}
May 25 13:05:54.463951 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.463899    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-684s7" event={"ID":"0f3d6ddb-a174-491b-b713-b4552899da35","Type":"ContainerStarted","Data":"d309b900daf0104fbd73aef1c2b25928b92351646083920a992e8ec60a7bf6e7"}
May 25 13:05:54.463951 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.463912    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-684s7" event={"ID":"0f3d6ddb-a174-491b-b713-b4552899da35","Type":"ContainerStarted","Data":"72db04ab07f9a7137c4ec5fd0456241bbc8746d06940ee29dd54620f87f9c7c0"}
May 25 13:05:54.465008 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.464980    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rckll" event={"ID":"d5eaa3b3-e7d0-466d-8e08-0e887a745119","Type":"ContainerStarted","Data":"a864c90525c228c56a9a988d81451b1395f315535235b383de738808e00e4acb"}
May 25 13:05:54.466210 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.466183    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tk4lc" event={"ID":"8269a18d-eb2f-4c7c-b747-dac1505afaba","Type":"ContainerStarted","Data":"d9755004c621edb5d13d709b5924e0e8ea88f7d753f21057f0d7b9048285ef70"}
May 25 13:05:54.467584 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.467559    2573 generic.go:358] "Generic (PLEG): container finished" podID="cda7f987-514d-47c2-b386-52b0eba3d858" containerID="d2f189cdcc3b772a89cbce3b8d2e6b22a845e382843af837585b02b43d09d5a5" exitCode=0
May 25 13:05:54.467669 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.467579    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mkkfs" event={"ID":"cda7f987-514d-47c2-b386-52b0eba3d858","Type":"ContainerDied","Data":"d2f189cdcc3b772a89cbce3b8d2e6b22a845e382843af837585b02b43d09d5a5"}
May 25 13:05:54.469156 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.469132    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt" event={"ID":"b1a80a50-dca4-426d-965b-6859348c8735","Type":"ContainerStarted","Data":"eefed48eaaef8567474eefaf5b21578bb214972fdd6071ba503c31a646c0406b"}
May 25 13:05:54.470280 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.470261    2573 generic.go:358] "Generic (PLEG): container finished" podID="dcd208f0122d6cb3e226ed802a3b1ab4" containerID="5327fb25f134e703a4fdf654e2b4ee7cb612bd8f44b905a0cb8b67b6e3b252e0" exitCode=0
May 25 13:05:54.470344 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.470328    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal" event={"ID":"dcd208f0122d6cb3e226ed802a3b1ab4","Type":"ContainerDied","Data":"5327fb25f134e703a4fdf654e2b4ee7cb612bd8f44b905a0cb8b67b6e3b252e0"}
May 25 13:05:54.471672 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.471641    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f78nn" event={"ID":"880fa08b-8cfb-45ce-8c9c-b63dd8896b39","Type":"ContainerStarted","Data":"7bdadf57f58d0f9869c6e01d7d18aa3297c54fbe4ff2f83561565142fbe3bf95"}
May 25 13:05:54.473150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.473130    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6zsvv" event={"ID":"0fd7df4c-9e76-4e26-9e43-68f9a7fa05fb","Type":"ContainerStarted","Data":"f17a8e5d283bd87f1630cf015d9b9e6f9f36a9f8d66c4737d96ad8ec37758ede"}
May 25 13:05:54.476098 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.476053    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c65mm" podStartSLOduration=2.970670078 podStartE2EDuration="21.476039769s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="2026-05-25 13:05:34.616192356 +0000 UTC m=+1.877795178" lastFinishedPulling="2026-05-25 13:05:53.121562046 +0000 UTC m=+20.383164869" observedRunningTime="2026-05-25 13:05:54.475711798 +0000 UTC m=+21.737314629" watchObservedRunningTime="2026-05-25 13:05:54.476039769 +0000 UTC m=+21.737642601"
May 25 13:05:54.488599 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.488532    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-rckll" podStartSLOduration=2.605700471 podStartE2EDuration="21.488521202s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="2026-05-25 13:05:34.570374711 +0000 UTC m=+1.831977520" lastFinishedPulling="2026-05-25 13:05:53.453195441 +0000 UTC m=+20.714798251" observedRunningTime="2026-05-25 13:05:54.488071562 +0000 UTC m=+21.749674396" watchObservedRunningTime="2026-05-25 13:05:54.488521202 +0000 UTC m=+21.750124033"
May 25 13:05:54.528010 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.527975    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6zsvv" podStartSLOduration=2.997046981 podStartE2EDuration="21.527965048s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="2026-05-25 13:05:34.632963595 +0000 UTC m=+1.894566405" lastFinishedPulling="2026-05-25 13:05:53.163881645 +0000 UTC m=+20.425484472" observedRunningTime="2026-05-25 13:05:54.527526549 +0000 UTC m=+21.789129392" watchObservedRunningTime="2026-05-25 13:05:54.527965048 +0000 UTC m=+21.789567879"
May 25 13:05:54.554904 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:54.554855    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f78nn" podStartSLOduration=3.033069593 podStartE2EDuration="21.554838017s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="2026-05-25 13:05:34.638561613 +0000 UTC m=+1.900164421" lastFinishedPulling="2026-05-25 13:05:53.160330021 +0000 UTC m=+20.421932845" observedRunningTime="2026-05-25 13:05:54.538915505 +0000 UTC m=+21.800518335" watchObservedRunningTime="2026-05-25 13:05:54.554838017 +0000 UTC m=+21.816440849"
May 25 13:05:55.152925 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.152905    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 13:05:55.296512 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.296418    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T13:05:55.15292094Z","UUID":"4a17c6d9-96aa-4623-b294-2875362818fe","Handler":null,"Name":"","Endpoint":""}
May 25 13:05:55.298899 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.298882    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 13:05:55.298989 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.298907    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 13:05:55.332956 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.332905    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:55.333034 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:55.332987    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:55.412030 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.412006    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:55.481316 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.481286    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-684s7" event={"ID":"0f3d6ddb-a174-491b-b713-b4552899da35","Type":"ContainerStarted","Data":"a362fe73af8fdcbe0214583644e3f2ea12c50271759bf2a7e3b68f7d6d3a2da2"}
May 25 13:05:55.482972 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.482932    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt" event={"ID":"b1a80a50-dca4-426d-965b-6859348c8735","Type":"ContainerStarted","Data":"96e0cae0dcf210cca4affed067c131e78b3cb815f3d1672c0148776b025254a6"}
May 25 13:05:55.484696 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.484672    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal" event={"ID":"dcd208f0122d6cb3e226ed802a3b1ab4","Type":"ContainerStarted","Data":"8af0aa602eb2c955eaf7ffa002ab3782848aaa5bf2c3b1a557cda76ee265d4d4"}
May 25 13:05:55.497066 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.497026    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal" podStartSLOduration=21.497015496 podStartE2EDuration="21.497015496s" podCreationTimestamp="2026-05-25 13:05:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:05:55.496930447 +0000 UTC m=+22.758533277" watchObservedRunningTime="2026-05-25 13:05:55.497015496 +0000 UTC m=+22.758618328"
May 25 13:05:55.497299 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:55.497277    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tk4lc" podStartSLOduration=3.9814749430000003 podStartE2EDuration="22.497271373s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="2026-05-25 13:05:34.644499765 +0000 UTC m=+1.906102577" lastFinishedPulling="2026-05-25 13:05:53.160296193 +0000 UTC m=+20.421899007" observedRunningTime="2026-05-25 13:05:54.554186501 +0000 UTC m=+21.815789332" watchObservedRunningTime="2026-05-25 13:05:55.497271373 +0000 UTC m=+22.758874230"
May 25 13:05:56.333466 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:56.333401    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:56.333606 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:56.333401    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:56.333606 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:56.333495    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:56.333681 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:56.333607    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:56.488379 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:56.488339    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt" event={"ID":"b1a80a50-dca4-426d-965b-6859348c8735","Type":"ContainerStarted","Data":"b8ae28556a7f9acb40eab6749cffbfd3bc510fe0b60ee76e9a2b29ce5b839732"}
May 25 13:05:56.518744 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:56.518699    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fxmt" podStartSLOduration=2.008173857 podStartE2EDuration="23.518681047s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="2026-05-25 13:05:34.549227169 +0000 UTC m=+1.810829978" lastFinishedPulling="2026-05-25 13:05:56.059734335 +0000 UTC m=+23.321337168" observedRunningTime="2026-05-25 13:05:56.518410521 +0000 UTC m=+23.780013352" watchObservedRunningTime="2026-05-25 13:05:56.518681047 +0000 UTC m=+23.780283879"
May 25 13:05:56.665132 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:56.665062    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:56.665762 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:56.665738    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:57.333242 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:57.333056    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:57.333413 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:57.333314    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:05:57.494948 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:57.494912    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-684s7" event={"ID":"0f3d6ddb-a174-491b-b713-b4552899da35","Type":"ContainerStarted","Data":"26e3250b33766f2ef6620cb3ead0311a08d5dc6990fbecfe95cc7f98f1d04283"}
May 25 13:05:57.495537 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:57.495521    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c65mm"
May 25 13:05:58.333064 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:58.333033    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:05:58.333064 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:58.333046    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:05:58.333285 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:58.333145    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:05:58.333331 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:58.333278    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:05:59.333566 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:05:59.333516    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:05:59.333990 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:05:59.333634    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:06:00.333645 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.333418    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:06:00.334275 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.333487    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:06:00.334275 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:00.333688    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:06:00.334275 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:00.333780    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:06:00.503036 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.503009    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-684s7" event={"ID":"0f3d6ddb-a174-491b-b713-b4552899da35","Type":"ContainerStarted","Data":"88e20afb7fcd4f69db93ec9ea09ff700156ebe2e34ec4b5460d7f0676e175cef"}
May 25 13:06:00.503344 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.503322    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:06:00.503433 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.503354    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:06:00.503433 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.503367    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:06:00.504715 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.504679    2573 generic.go:358] "Generic (PLEG): container finished" podID="cda7f987-514d-47c2-b386-52b0eba3d858" containerID="ce0448c000c2d4f51f7c635a2ae416310c0651dec367808fab1c1172357ee4d3" exitCode=0
May 25 13:06:00.504827 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.504721    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mkkfs" event={"ID":"cda7f987-514d-47c2-b386-52b0eba3d858","Type":"ContainerDied","Data":"ce0448c000c2d4f51f7c635a2ae416310c0651dec367808fab1c1172357ee4d3"}
May 25 13:06:00.519029 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.519010    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:06:00.519196 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.519183    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:06:00.530285 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:00.530251    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-684s7" podStartSLOduration=8.177133017 podStartE2EDuration="27.530241985s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="2026-05-25 13:05:34.592106842 +0000 UTC m=+1.853709650" lastFinishedPulling="2026-05-25 13:05:53.945215795 +0000 UTC m=+21.206818618" observedRunningTime="2026-05-25 13:06:00.529921321 +0000 UTC m=+27.791524151" watchObservedRunningTime="2026-05-25 13:06:00.530241985 +0000 UTC m=+27.791844816"
May 25 13:06:01.332752 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:01.332728    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:06:01.332896 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:01.332842    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:06:01.508459 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:01.508396    2573 generic.go:358] "Generic (PLEG): container finished" podID="cda7f987-514d-47c2-b386-52b0eba3d858" containerID="26eed913473f5a7359d5c9b3085457675f1b2a867c23cdef33378df9c94f08d6" exitCode=0
May 25 13:06:01.508806 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:01.508477    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mkkfs" event={"ID":"cda7f987-514d-47c2-b386-52b0eba3d858","Type":"ContainerDied","Data":"26eed913473f5a7359d5c9b3085457675f1b2a867c23cdef33378df9c94f08d6"}
May 25 13:06:01.619872 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:01.619846    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-92l8w"]
May 25 13:06:01.619976 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:01.619963    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:06:01.620051 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:01.620035    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:06:01.623839 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:01.623816    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tkljt"]
May 25 13:06:01.623974 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:01.623929    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:06:01.624058 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:01.624029    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:06:01.624641 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:01.624613    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7xhlm"]
May 25 13:06:01.624731 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:01.624689    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:06:01.624774 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:01.624763    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:06:02.512200 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:02.512126    2573 generic.go:358] "Generic (PLEG): container finished" podID="cda7f987-514d-47c2-b386-52b0eba3d858" containerID="b495cc2f12b12cbfb62f2e177df8391a6acd9c11a7095b23abdf95c51099ade6" exitCode=0
May 25 13:06:02.512584 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:02.512205    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mkkfs" event={"ID":"cda7f987-514d-47c2-b386-52b0eba3d858","Type":"ContainerDied","Data":"b495cc2f12b12cbfb62f2e177df8391a6acd9c11a7095b23abdf95c51099ade6"}
May 25 13:06:03.333355 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:03.333318    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:06:03.333518 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:03.333415    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:06:03.333518 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:03.333448    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:06:03.333655 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:03.333563    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:06:03.333655 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:03.333567    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:06:03.333655 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:03.333641    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:06:05.332973 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:05.332796    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:06:05.333336 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:05.332803    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:06:05.333336 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:05.333051    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkljt" podUID="c1774200-9350-464d-a3b8-2be8812bcde4"
May 25 13:06:05.333336 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:05.333146    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l8w" podUID="430200e3-29df-46e6-99ce-346a015400ae"
May 25 13:06:05.333336 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:05.332803    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:06:05.333336 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:05.333239    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xhlm" podUID="bd26de80-992a-449c-b62d-608a45a4fa12"
May 25 13:06:06.089701 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.089666    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:06:06.089906 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.089730    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:06:06.089906 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.089819    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:06:06.089906 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.089854    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:06:06.089906 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.089884    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs podName:c1774200-9350-464d-a3b8-2be8812bcde4 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.089865449 +0000 UTC m=+65.351468274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs") pod "network-metrics-daemon-tkljt" (UID: "c1774200-9350-464d-a3b8-2be8812bcde4") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:06:06.089906 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.089908    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret podName:430200e3-29df-46e6-99ce-346a015400ae nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.08989167 +0000 UTC m=+65.351494511 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret") pod "global-pull-secret-syncer-92l8w" (UID: "430200e3-29df-46e6-99ce-346a015400ae") : object "kube-system"/"original-pull-secret" not registered
May 25 13:06:06.190051 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.190016    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6sj\" (UniqueName: \"kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj\") pod \"network-check-target-7xhlm\" (UID: \"bd26de80-992a-449c-b62d-608a45a4fa12\") " pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:06:06.190207 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.190172    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:06:06.190207 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.190192    2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:06:06.190207 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.190205    2573 projected.go:194] Error preparing data for projected volume kube-api-access-4m6sj for pod openshift-network-diagnostics/network-check-target-7xhlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:06:06.190334 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.190254    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj podName:bd26de80-992a-449c-b62d-608a45a4fa12 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.190238813 +0000 UTC m=+65.451841641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m6sj" (UniqueName: "kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj") pod "network-check-target-7xhlm" (UID: "bd26de80-992a-449c-b62d-608a45a4fa12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:06:06.499212 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.499193    2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeReady"
May 25 13:06:06.499625 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.499281    2573 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 13:06:06.532263 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.532240    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"]
May 25 13:06:06.549473 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.549444    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fb44d44c7-qzx96"]
May 25 13:06:06.549629 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.549606    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:06.553320 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.552456    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 13:06:06.553320 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.552756    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 25 13:06:06.553320 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.552888    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 25 13:06:06.553320 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.553012    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-96p87\""
May 25 13:06:06.553320 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.553102    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 13:06:06.565708 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.565689    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6"]
May 25 13:06:06.565839 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.565824    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.569870 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.569836    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lntzn\""
May 25 13:06:06.570159 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.570133    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 13:06:06.570159 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.570133    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 13:06:06.570398 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.570362    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 13:06:06.581302 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.581283    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 13:06:06.583943 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.583920    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-84964b69b-gj5s2"]
May 25 13:06:06.584071 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.584051    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6"
May 25 13:06:06.586663 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.586643    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 25 13:06:06.586766 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.586720    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-57h78\""
May 25 13:06:06.587047 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.587032    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 25 13:06:06.601326 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.601310    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"]
May 25 13:06:06.601463 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.601443    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.604288 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.604269    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 25 13:06:06.604623 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.604607    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 25 13:06:06.604943 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.604921    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6ch8m\""
May 25 13:06:06.605040 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.604992    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 25 13:06:06.605040 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.604999    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 25 13:06:06.605040 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.605021    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 25 13:06:06.612498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.612483    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 25 13:06:06.613634 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.613615    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"]
May 25 13:06:06.613802 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.613782    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:06.617036 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.617015    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fz5sg\""
May 25 13:06:06.618273 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.618218    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 25 13:06:06.618273 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.618249    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 25 13:06:06.618502 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.618487    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 25 13:06:06.632216 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.632196    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-sqk9d"]
May 25 13:06:06.632355 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.632338    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.635043 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.635028    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 25 13:06:06.635517 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.635328    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 25 13:06:06.635517 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.635397    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 25 13:06:06.635517 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.635451    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 25 13:06:06.635517 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.635470    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bnlxr\""
May 25 13:06:06.650375 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.650351    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"]
May 25 13:06:06.650503 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.650485    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.655241 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.655220    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 13:06:06.655390 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.655371    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 25 13:06:06.655594 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.655576    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 13:06:06.655684 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.655611    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pk972\""
May 25 13:06:06.656310 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.656286    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 25 13:06:06.662494 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.662475    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 25 13:06:06.663753 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.663530    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-jwb2k"]
May 25 13:06:06.663753 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.663708    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.666672 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.666648    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 25 13:06:06.666920 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.666862    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8kng4\""
May 25 13:06:06.667012 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.666957    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 25 13:06:06.667012 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.667005    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 25 13:06:06.667123 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.667103    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 25 13:06:06.684193 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.684119    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"]
May 25 13:06:06.684290 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.684215    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-sqk9d"]
May 25 13:06:06.684290 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.684224    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.684290 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.684233    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"]
May 25 13:06:06.684290 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.684248    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr"]
May 25 13:06:06.686845 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.686826    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 25 13:06:06.686845 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.686838    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 25 13:06:06.687034 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.686851    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fchg4\""
May 25 13:06:06.687034 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.686859    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 25 13:06:06.687233 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.687213    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 25 13:06:06.694041 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694019    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrdw\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-kube-api-access-bcrdw\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.694142 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694067    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52fc4\" (UniqueName: \"kubernetes.io/projected/a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8-kube-api-access-52fc4\") pod \"kube-storage-version-migrator-operator-565644f49-8kgql\" (UID: \"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.694202 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694160    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/080f2207-b1de-4749-ab1d-a8e445abf8fa-ca-trust-extracted\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.694202 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694184    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 25 13:06:06.694202 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694194    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.694360 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694221    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:06.694360 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.694360 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.694510 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8-config\") pod \"kube-storage-version-migrator-operator-565644f49-8kgql\" (UID: \"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.694510 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694424    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-bound-sa-token\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.694510 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694442    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:06.694510 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694472    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-default-certificate\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.694510 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgtlp\" (UniqueName: \"kubernetes.io/projected/21ef9931-fc75-4841-b015-3a450c700b77-kube-api-access-wgtlp\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.694510 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694506    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6e7fa479-726d-460c-b125-97d1adafa418-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:06.694791 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694526    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-certificates\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.694791 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694576    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-trusted-ca\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.694791 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694607    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-installation-pull-secrets\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.694791 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694630    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47cjs\" (UniqueName: \"kubernetes.io/projected/5eda4f74-db07-4c34-af65-2136bf98fe2e-kube-api-access-47cjs\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:06.694791 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694662    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v29t4\" (UniqueName: \"kubernetes.io/projected/6e7fa479-726d-460c-b125-97d1adafa418-kube-api-access-v29t4\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:06.694791 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694702    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-image-registry-private-configuration\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.694791 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-8kgql\" (UID: \"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.694791 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694745    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvj47\" (UniqueName: \"kubernetes.io/projected/05c02256-ea7d-4a74-8f08-a16384769838-kube-api-access-nvj47\") pod \"volume-data-source-validator-5cf8c89d64-ffjg6\" (UID: \"05c02256-ea7d-4a74-8f08-a16384769838\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6"
May 25 13:06:06.695092 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.694824    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-stats-auth\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.705587 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.705568    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"]
May 25 13:06:06.705669 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.705602    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-jwb2k"]
May 25 13:06:06.705669 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.705607    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr"
May 25 13:06:06.705754 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.705614    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb44d44c7-qzx96"]
May 25 13:06:06.705754 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.705749    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84964b69b-gj5s2"]
May 25 13:06:06.705820 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.705764    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hjzcb"]
May 25 13:06:06.707987 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.707964    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 13:06:06.708086 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.707966    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-87mrw\""
May 25 13:06:06.708086 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.707965    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 13:06:06.725815 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.725792    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6"]
May 25 13:06:06.725918 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.725824    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr"]
May 25 13:06:06.725918 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.725837    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"]
May 25 13:06:06.725918 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.725851    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjzcb"]
May 25 13:06:06.725918 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.725891    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q6pgt"]
May 25 13:06:06.726695 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.726675    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:06.729361 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.729335    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 13:06:06.729455 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.729378    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vbzlk\""
May 25 13:06:06.729455 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.729444    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 13:06:06.729602 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.729464    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 13:06:06.743063 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.743045    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q6pgt"]
May 25 13:06:06.743163 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.743141    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:06.745288 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.745271    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 13:06:06.745388 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.745290    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 13:06:06.745388 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.745348    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 13:06:06.745388 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.745362    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5d4pm\""
May 25 13:06:06.745388 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.745271    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 13:06:06.795807 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.795729    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrdw\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-kube-api-access-bcrdw\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.795807 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.795775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:06.795968 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.795807    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52fc4\" (UniqueName: \"kubernetes.io/projected/a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8-kube-api-access-52fc4\") pod \"kube-storage-version-migrator-operator-565644f49-8kgql\" (UID: \"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.795968 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.795838    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s55hk\" (UniqueName: \"kubernetes.io/projected/2f58bc38-d685-405e-aa7e-7de7a86001ee-kube-api-access-s55hk\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:06.795968 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.795901    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:06.795968 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.795926    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.795968 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.795957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-snapshots\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.796213 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.796172    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:06.796261 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.796238    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls podName:6e7fa479-726d-460c-b125-97d1adafa418 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:07.296220845 +0000 UTC m=+34.557823656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-j54sp" (UID: "6e7fa479-726d-460c-b125-97d1adafa418") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:06.796327 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.796301    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:06.796327 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796315    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.796509 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.796334    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:07.296322894 +0000 UTC m=+34.557925710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : secret "router-metrics-certs-default" not found
May 25 13:06:06.796509 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796367    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/127a81c2-9752-4c7f-979c-d5aa0d2a86b5-config\") pod \"service-ca-operator-57b5b68b8-vc5f4\" (UID: \"127a81c2-9752-4c7f-979c-d5aa0d2a86b5\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.796509 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796397    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-trusted-ca\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.796509 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkl87\" (UniqueName: \"kubernetes.io/projected/127a81c2-9752-4c7f-979c-d5aa0d2a86b5-kube-api-access-mkl87\") pod \"service-ca-operator-57b5b68b8-vc5f4\" (UID: \"127a81c2-9752-4c7f-979c-d5aa0d2a86b5\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.796509 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796451    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:06.796509 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796483    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-47cjs\" (UniqueName: \"kubernetes.io/projected/5eda4f74-db07-4c34-af65-2136bf98fe2e-kube-api-access-47cjs\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796510    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v29t4\" (UniqueName: \"kubernetes.io/projected/6e7fa479-726d-460c-b125-97d1adafa418-kube-api-access-v29t4\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796539    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-image-registry-private-configuration\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796581    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-8kgql\" (UID: \"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796608    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvj47\" (UniqueName: \"kubernetes.io/projected/05c02256-ea7d-4a74-8f08-a16384769838-kube-api-access-nvj47\") pod \"volume-data-source-validator-5cf8c89d64-ffjg6\" (UID: \"05c02256-ea7d-4a74-8f08-a16384769838\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796644    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-stats-auth\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796670    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-serving-cert\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796696    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldvnr\" (UniqueName: \"kubernetes.io/projected/f3626d2c-129b-4a27-9855-4d9649bf3c0f-kube-api-access-ldvnr\") pod \"network-check-source-7f44b6dd87-dlmmr\" (UID: \"f3626d2c-129b-4a27-9855-4d9649bf3c0f\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796722    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/127a81c2-9752-4c7f-979c-d5aa0d2a86b5-serving-cert\") pod \"service-ca-operator-57b5b68b8-vc5f4\" (UID: \"127a81c2-9752-4c7f-979c-d5aa0d2a86b5\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796755    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/080f2207-b1de-4749-ab1d-a8e445abf8fa-ca-trust-extracted\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshjg\" (UniqueName: \"kubernetes.io/projected/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-kube-api-access-rshjg\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796808    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-tmp\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.796844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796847    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-config\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796876    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-service-ca-bundle\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796934    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.796960    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8-config\") pod \"kube-storage-version-migrator-operator-565644f49-8kgql\" (UID: \"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.797002    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-bound-sa-token\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.797026    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-default-certificate\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.797051    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtlp\" (UniqueName: \"kubernetes.io/projected/21ef9931-fc75-4841-b015-3a450c700b77-kube-api-access-wgtlp\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.797080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sw56\" (UniqueName: \"kubernetes.io/projected/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-kube-api-access-6sw56\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.797115    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6e7fa479-726d-460c-b125-97d1adafa418-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.797142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-serving-cert\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.797175    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-certificates\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.797201    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-trusted-ca\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.797354 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.797226    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-installation-pull-secrets\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.797949 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.797768    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:07.297751218 +0000 UTC m=+34.559354047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : configmap references non-existent config key: service-ca.crt
May 25 13:06:06.797949 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.797870    2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:06.797949 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.797884    2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb44d44c7-qzx96: secret "image-registry-tls" not found
May 25 13:06:06.797949 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.797920    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls podName:080f2207-b1de-4749-ab1d-a8e445abf8fa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:07.297906224 +0000 UTC m=+34.559509052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls") pod "image-registry-fb44d44c7-qzx96" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa") : secret "image-registry-tls" not found
May 25 13:06:06.797949 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.797922    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/080f2207-b1de-4749-ab1d-a8e445abf8fa-ca-trust-extracted\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.798209 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.798190    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:06.798271 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.798246    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls podName:5eda4f74-db07-4c34-af65-2136bf98fe2e nodeName:}" failed. No retries permitted until 2026-05-25 13:06:07.29822995 +0000 UTC m=+34.559832774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls") pod "cluster-samples-operator-697587599-q5h6p" (UID: "5eda4f74-db07-4c34-af65-2136bf98fe2e") : secret "samples-operator-tls" not found
May 25 13:06:06.799183 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.799130    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8-config\") pod \"kube-storage-version-migrator-operator-565644f49-8kgql\" (UID: \"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.802242 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.802218    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-installation-pull-secrets\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.802324 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.802305    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-default-certificate\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.802739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.802717    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-8kgql\" (UID: \"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.802739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.802729    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-image-registry-private-configuration\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.802985 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.802965    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-stats-auth\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.803932 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.803914    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52fc4\" (UniqueName: \"kubernetes.io/projected/a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8-kube-api-access-52fc4\") pod \"kube-storage-version-migrator-operator-565644f49-8kgql\" (UID: \"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.806774 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.806731    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtlp\" (UniqueName: \"kubernetes.io/projected/21ef9931-fc75-4841-b015-3a450c700b77-kube-api-access-wgtlp\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:06.806864 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.806840    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvj47\" (UniqueName: \"kubernetes.io/projected/05c02256-ea7d-4a74-8f08-a16384769838-kube-api-access-nvj47\") pod \"volume-data-source-validator-5cf8c89d64-ffjg6\" (UID: \"05c02256-ea7d-4a74-8f08-a16384769838\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6"
May 25 13:06:06.806964 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.806940    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47cjs\" (UniqueName: \"kubernetes.io/projected/5eda4f74-db07-4c34-af65-2136bf98fe2e-kube-api-access-47cjs\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:06.808053 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.808032    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-certificates\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.808133 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.808039    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6e7fa479-726d-460c-b125-97d1adafa418-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:06.808437 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.808395    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-trusted-ca\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.810187 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.810154    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v29t4\" (UniqueName: \"kubernetes.io/projected/6e7fa479-726d-460c-b125-97d1adafa418-kube-api-access-v29t4\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:06.810187 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.810168    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-bound-sa-token\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.810329 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.810304    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrdw\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-kube-api-access-bcrdw\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:06.894279 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.894257    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6"
May 25 13:06:06.898130 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898110    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-serving-cert\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.898242 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898137    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ldvnr\" (UniqueName: \"kubernetes.io/projected/f3626d2c-129b-4a27-9855-4d9649bf3c0f-kube-api-access-ldvnr\") pod \"network-check-source-7f44b6dd87-dlmmr\" (UID: \"f3626d2c-129b-4a27-9855-4d9649bf3c0f\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr"
May 25 13:06:06.898242 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898156    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/127a81c2-9752-4c7f-979c-d5aa0d2a86b5-serving-cert\") pod \"service-ca-operator-57b5b68b8-vc5f4\" (UID: \"127a81c2-9752-4c7f-979c-d5aa0d2a86b5\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.898242 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898178    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rshjg\" (UniqueName: \"kubernetes.io/projected/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-kube-api-access-rshjg\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.898624 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898448    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-tmp\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.898624 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898504    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-config\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.898624 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898572    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-service-ca-bundle\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.898817 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898638    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6sw56\" (UniqueName: \"kubernetes.io/projected/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-kube-api-access-6sw56\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.898817 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898668    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-serving-cert\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.898817 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898695    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2r8\" (UniqueName: \"kubernetes.io/projected/e0e32442-e801-4fcb-8c48-74b788e18965-kube-api-access-gw2r8\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:06.898817 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898726    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0e32442-e801-4fcb-8c48-74b788e18965-tmp-dir\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:06.898817 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898778    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:06.898817 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898794    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-tmp\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.898817 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898816    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s55hk\" (UniqueName: \"kubernetes.io/projected/2f58bc38-d685-405e-aa7e-7de7a86001ee-kube-api-access-s55hk\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:06.899150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898895    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-snapshots\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.899150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898928    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.899150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898953    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/127a81c2-9752-4c7f-979c-d5aa0d2a86b5-config\") pod \"service-ca-operator-57b5b68b8-vc5f4\" (UID: \"127a81c2-9752-4c7f-979c-d5aa0d2a86b5\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.899150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.898980    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-trusted-ca\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.899150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.899006    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mkl87\" (UniqueName: \"kubernetes.io/projected/127a81c2-9752-4c7f-979c-d5aa0d2a86b5-kube-api-access-mkl87\") pod \"service-ca-operator-57b5b68b8-vc5f4\" (UID: \"127a81c2-9752-4c7f-979c-d5aa0d2a86b5\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.899150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.899052    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:06.899150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.899088    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e32442-e801-4fcb-8c48-74b788e18965-config-volume\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:06.899473 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.899212    2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:06:06.899473 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.899217    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-config\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.899473 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:06.899272    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert podName:2f58bc38-d685-405e-aa7e-7de7a86001ee nodeName:}" failed. No retries permitted until 2026-05-25 13:06:07.399255146 +0000 UTC m=+34.660857968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert") pod "ingress-canary-hjzcb" (UID: "2f58bc38-d685-405e-aa7e-7de7a86001ee") : secret "canary-serving-cert" not found
May 25 13:06:06.899473 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.899350    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-service-ca-bundle\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.899839 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.899763    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/127a81c2-9752-4c7f-979c-d5aa0d2a86b5-config\") pod \"service-ca-operator-57b5b68b8-vc5f4\" (UID: \"127a81c2-9752-4c7f-979c-d5aa0d2a86b5\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.900068 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.900044    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-snapshots\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.900213 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.900196    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-trusted-ca\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.900349 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.900325    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.900689 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.900665    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/127a81c2-9752-4c7f-979c-d5aa0d2a86b5-serving-cert\") pod \"service-ca-operator-57b5b68b8-vc5f4\" (UID: \"127a81c2-9752-4c7f-979c-d5aa0d2a86b5\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.900777 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.900674    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-serving-cert\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.901204 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.901186    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-serving-cert\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.906117 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.906075    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshjg\" (UniqueName: \"kubernetes.io/projected/ca0dc551-06fa-414c-a3e3-28f8d0b2e382-kube-api-access-rshjg\") pod \"console-operator-76dff48d59-jwb2k\" (UID: \"ca0dc551-06fa-414c-a3e3-28f8d0b2e382\") " pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:06.906331 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.906310    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sw56\" (UniqueName: \"kubernetes.io/projected/a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59-kube-api-access-6sw56\") pod \"insights-operator-78bdfb888c-sqk9d\" (UID: \"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59\") " pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.906415 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.906344    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldvnr\" (UniqueName: \"kubernetes.io/projected/f3626d2c-129b-4a27-9855-4d9649bf3c0f-kube-api-access-ldvnr\") pod \"network-check-source-7f44b6dd87-dlmmr\" (UID: \"f3626d2c-129b-4a27-9855-4d9649bf3c0f\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr"
May 25 13:06:06.909413 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.909390    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s55hk\" (UniqueName: \"kubernetes.io/projected/2f58bc38-d685-405e-aa7e-7de7a86001ee-kube-api-access-s55hk\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:06.909500 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.909451    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkl87\" (UniqueName: \"kubernetes.io/projected/127a81c2-9752-4c7f-979c-d5aa0d2a86b5-kube-api-access-mkl87\") pod \"service-ca-operator-57b5b68b8-vc5f4\" (UID: \"127a81c2-9752-4c7f-979c-d5aa0d2a86b5\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.941407 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.941386    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"
May 25 13:06:06.962236 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.962215    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-sqk9d"
May 25 13:06:06.973865 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.973845    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"
May 25 13:06:06.994445 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:06.994422    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:07.000138 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.000115    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e32442-e801-4fcb-8c48-74b788e18965-config-volume\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:07.000234 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.000205    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2r8\" (UniqueName: \"kubernetes.io/projected/e0e32442-e801-4fcb-8c48-74b788e18965-kube-api-access-gw2r8\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:07.000234 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.000230    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0e32442-e801-4fcb-8c48-74b788e18965-tmp-dir\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:07.000351 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.000336    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:07.000442 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.000426    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:06:07.000506 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.000491    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls podName:e0e32442-e801-4fcb-8c48-74b788e18965 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:07.500476032 +0000 UTC m=+34.762078855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls") pod "dns-default-q6pgt" (UID: "e0e32442-e801-4fcb-8c48-74b788e18965") : secret "dns-default-metrics-tls" not found
May 25 13:06:07.006794 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.006773    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0e32442-e801-4fcb-8c48-74b788e18965-tmp-dir\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:07.006934 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.006915    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e32442-e801-4fcb-8c48-74b788e18965-config-volume\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:07.012884 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.012861    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2r8\" (UniqueName: \"kubernetes.io/projected/e0e32442-e801-4fcb-8c48-74b788e18965-kube-api-access-gw2r8\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:07.015730 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.015714    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr"
May 25 13:06:07.031053 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.031028    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-wfc8p"]
May 25 13:06:07.056795 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.056745    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.059136 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.059115    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wmpmm\""
May 25 13:06:07.201560 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.201508    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7cq9\" (UniqueName: \"kubernetes.io/projected/ab2520dc-3572-44d2-b557-7d15a0bc0c45-kube-api-access-q7cq9\") pod \"node-resolver-wfc8p\" (UID: \"ab2520dc-3572-44d2-b557-7d15a0bc0c45\") " pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.201726 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.201650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2520dc-3572-44d2-b557-7d15a0bc0c45-tmp-dir\") pod \"node-resolver-wfc8p\" (UID: \"ab2520dc-3572-44d2-b557-7d15a0bc0c45\") " pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.201794 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.201726    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab2520dc-3572-44d2-b557-7d15a0bc0c45-hosts-file\") pod \"node-resolver-wfc8p\" (UID: \"ab2520dc-3572-44d2-b557-7d15a0bc0c45\") " pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.302219 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.302188    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:07.302219 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.302224    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:07.302431 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.302254    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cq9\" (UniqueName: \"kubernetes.io/projected/ab2520dc-3572-44d2-b557-7d15a0bc0c45-kube-api-access-q7cq9\") pod \"node-resolver-wfc8p\" (UID: \"ab2520dc-3572-44d2-b557-7d15a0bc0c45\") " pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.302431 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.302280    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:07.302431 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.302331    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2520dc-3572-44d2-b557-7d15a0bc0c45-tmp-dir\") pod \"node-resolver-wfc8p\" (UID: \"ab2520dc-3572-44d2-b557-7d15a0bc0c45\") " pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.302431 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.302340    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:07.302431 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.302391    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:07.302431 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.302402    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls podName:6e7fa479-726d-460c-b125-97d1adafa418 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:08.302384034 +0000 UTC m=+35.563986850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-j54sp" (UID: "6e7fa479-726d-460c-b125-97d1adafa418") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:07.302431 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.302338    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.302448    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls podName:5eda4f74-db07-4c34-af65-2136bf98fe2e nodeName:}" failed. No retries permitted until 2026-05-25 13:06:08.302432063 +0000 UTC m=+35.564034891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls") pod "cluster-samples-operator-697587599-q5h6p" (UID: "5eda4f74-db07-4c34-af65-2136bf98fe2e") : secret "samples-operator-tls" not found
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.302473    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:08.302461227 +0000 UTC m=+35.564064043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : secret "router-metrics-certs-default" not found
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.302591    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab2520dc-3572-44d2-b557-7d15a0bc0c45-hosts-file\") pod \"node-resolver-wfc8p\" (UID: \"ab2520dc-3572-44d2-b557-7d15a0bc0c45\") " pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.302638    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.302657    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2520dc-3572-44d2-b557-7d15a0bc0c45-tmp-dir\") pod \"node-resolver-wfc8p\" (UID: \"ab2520dc-3572-44d2-b557-7d15a0bc0c45\") " pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.302671    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.302677    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab2520dc-3572-44d2-b557-7d15a0bc0c45-hosts-file\") pod \"node-resolver-wfc8p\" (UID: \"ab2520dc-3572-44d2-b557-7d15a0bc0c45\") " pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.302748    2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.302757    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:08.302744852 +0000 UTC m=+35.564347665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : configmap references non-existent config key: service-ca.crt
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.302761    2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb44d44c7-qzx96: secret "image-registry-tls" not found
May 25 13:06:07.302804 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.302803    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls podName:080f2207-b1de-4749-ab1d-a8e445abf8fa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:08.302789594 +0000 UTC m=+35.564392405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls") pod "image-registry-fb44d44c7-qzx96" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa") : secret "image-registry-tls" not found
May 25 13:06:07.310258 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.310199    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cq9\" (UniqueName: \"kubernetes.io/projected/ab2520dc-3572-44d2-b557-7d15a0bc0c45-kube-api-access-q7cq9\") pod \"node-resolver-wfc8p\" (UID: \"ab2520dc-3572-44d2-b557-7d15a0bc0c45\") " pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.336212 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.336192    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:06:07.336212 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.336211    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:06:07.336371 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.336304    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:06:07.338605 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.338474    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 13:06:07.338605 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.338517    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcddr\""
May 25 13:06:07.338605 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.338527    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-clhmx\""
May 25 13:06:07.338605 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.338572    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 13:06:07.368388 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.368367    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wfc8p"
May 25 13:06:07.403309 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.403286    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:07.403452 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.403424    2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:06:07.403564 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.403487    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert podName:2f58bc38-d685-405e-aa7e-7de7a86001ee nodeName:}" failed. No retries permitted until 2026-05-25 13:06:08.403471316 +0000 UTC m=+35.665074143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert") pod "ingress-canary-hjzcb" (UID: "2f58bc38-d685-405e-aa7e-7de7a86001ee") : secret "canary-serving-cert" not found
May 25 13:06:07.504307 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:07.504270    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:07.504907 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.504760    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:06:07.504907 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:07.504863    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls podName:e0e32442-e801-4fcb-8c48-74b788e18965 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:08.504848228 +0000 UTC m=+35.766451036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls") pod "dns-default-q6pgt" (UID: "e0e32442-e801-4fcb-8c48-74b788e18965") : secret "dns-default-metrics-tls" not found
May 25 13:06:08.311075 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.311031    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:08.311075 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.311072    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.311106    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.311175    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.311177    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.311204    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.311251    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls podName:6e7fa479-726d-460c-b125-97d1adafa418 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:10.311230227 +0000 UTC m=+37.572833037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-j54sp" (UID: "6e7fa479-726d-460c-b125-97d1adafa418") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.311273    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.311293    2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.311305    2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb44d44c7-qzx96: secret "image-registry-tls" not found
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.311278    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.311347    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls podName:080f2207-b1de-4749-ab1d-a8e445abf8fa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:10.311330885 +0000 UTC m=+37.572933713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls") pod "image-registry-fb44d44c7-qzx96" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa") : secret "image-registry-tls" not found
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.311363    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:10.3113545 +0000 UTC m=+37.572957324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : secret "router-metrics-certs-default" not found
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.311401    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:10.31139308 +0000 UTC m=+37.572995889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : configmap references non-existent config key: service-ca.crt
May 25 13:06:08.311406 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.311416    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls podName:5eda4f74-db07-4c34-af65-2136bf98fe2e nodeName:}" failed. No retries permitted until 2026-05-25 13:06:10.311408287 +0000 UTC m=+37.573011124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls") pod "cluster-samples-operator-697587599-q5h6p" (UID: "5eda4f74-db07-4c34-af65-2136bf98fe2e") : secret "samples-operator-tls" not found
May 25 13:06:08.411958 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.411922    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:08.412126 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.412084    2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:06:08.412195 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.412151    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert podName:2f58bc38-d685-405e-aa7e-7de7a86001ee nodeName:}" failed. No retries permitted until 2026-05-25 13:06:10.412133889 +0000 UTC m=+37.673736716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert") pod "ingress-canary-hjzcb" (UID: "2f58bc38-d685-405e-aa7e-7de7a86001ee") : secret "canary-serving-cert" not found
May 25 13:06:08.513494 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.513464    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:08.513919 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.513571    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:06:08.513919 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:08.513628    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls podName:e0e32442-e801-4fcb-8c48-74b788e18965 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:10.513609106 +0000 UTC m=+37.775211932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls") pod "dns-default-q6pgt" (UID: "e0e32442-e801-4fcb-8c48-74b788e18965") : secret "dns-default-metrics-tls" not found
May 25 13:06:08.735298 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:08.735264    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2520dc_3572_44d2_b557_7d15a0bc0c45.slice/crio-2f007d4a59192542f48a07e1b8c9e7c386bd0e5ffca8888b6c94b4d213fbb69f WatchSource:0}: Error finding container 2f007d4a59192542f48a07e1b8c9e7c386bd0e5ffca8888b6c94b4d213fbb69f: Status 404 returned error can't find the container with id 2f007d4a59192542f48a07e1b8c9e7c386bd0e5ffca8888b6c94b4d213fbb69f
May 25 13:06:08.972517 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.972487    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-sqk9d"]
May 25 13:06:08.975431 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.975382    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6"]
May 25 13:06:08.981730 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:08.981704    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a4f5bc_df6e_4bea_9f95_8ec41af6fb59.slice/crio-3f66a0c5cc7eeb24e373495655ece18745df8f4e3595b7346921ea16c1f627f5 WatchSource:0}: Error finding container 3f66a0c5cc7eeb24e373495655ece18745df8f4e3595b7346921ea16c1f627f5: Status 404 returned error can't find the container with id 3f66a0c5cc7eeb24e373495655ece18745df8f4e3595b7346921ea16c1f627f5
May 25 13:06:08.982364 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:08.982332    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c02256_ea7d_4a74_8f08_a16384769838.slice/crio-61ca126f78c7e6cc124c48b5d6e7ac5f23ed3ef0e343cee048132c78d7677484 WatchSource:0}: Error finding container 61ca126f78c7e6cc124c48b5d6e7ac5f23ed3ef0e343cee048132c78d7677484: Status 404 returned error can't find the container with id 61ca126f78c7e6cc124c48b5d6e7ac5f23ed3ef0e343cee048132c78d7677484
May 25 13:06:08.984721 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.984698    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql"]
May 25 13:06:08.988430 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:08.988405    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bfc893_a8bd_4f6b_ab4b_adcdd82c96f8.slice/crio-7535c604fd133d6d81f68584f34449b62a0bd24c49dc30ed81f60d739a9b024f WatchSource:0}: Error finding container 7535c604fd133d6d81f68584f34449b62a0bd24c49dc30ed81f60d739a9b024f: Status 404 returned error can't find the container with id 7535c604fd133d6d81f68584f34449b62a0bd24c49dc30ed81f60d739a9b024f
May 25 13:06:08.988770 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.988753    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr"]
May 25 13:06:08.989638 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:08.989600    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4"]
May 25 13:06:08.991318 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:08.991298    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127a81c2_9752_4c7f_979c_d5aa0d2a86b5.slice/crio-4c36e4e5d55852befd66656f959bfcaee81d29a6f006063b580cb0bb80f99ca9 WatchSource:0}: Error finding container 4c36e4e5d55852befd66656f959bfcaee81d29a6f006063b580cb0bb80f99ca9: Status 404 returned error can't find the container with id 4c36e4e5d55852befd66656f959bfcaee81d29a6f006063b580cb0bb80f99ca9
May 25 13:06:08.994389 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:08.994369    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3626d2c_129b_4a27_9855_4d9649bf3c0f.slice/crio-cb49d0ac73dedd570b3ea015b0f41ee17b751d9e68b2562a82ec6ae4dbac7b76 WatchSource:0}: Error finding container cb49d0ac73dedd570b3ea015b0f41ee17b751d9e68b2562a82ec6ae4dbac7b76: Status 404 returned error can't find the container with id cb49d0ac73dedd570b3ea015b0f41ee17b751d9e68b2562a82ec6ae4dbac7b76
May 25 13:06:09.001731 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.001692    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-jwb2k"]
May 25 13:06:09.003874 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:09.003857    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0dc551_06fa_414c_a3e3_28f8d0b2e382.slice/crio-9a3b18af2bcd33e10976b8dc26525aee973de2213c329fca165d2ed4c4deae03 WatchSource:0}: Error finding container 9a3b18af2bcd33e10976b8dc26525aee973de2213c329fca165d2ed4c4deae03: Status 404 returned error can't find the container with id 9a3b18af2bcd33e10976b8dc26525aee973de2213c329fca165d2ed4c4deae03
May 25 13:06:09.526148 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.526113    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k" event={"ID":"ca0dc551-06fa-414c-a3e3-28f8d0b2e382","Type":"ContainerStarted","Data":"9a3b18af2bcd33e10976b8dc26525aee973de2213c329fca165d2ed4c4deae03"}
May 25 13:06:09.527246 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.527221    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sqk9d" event={"ID":"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59","Type":"ContainerStarted","Data":"3f66a0c5cc7eeb24e373495655ece18745df8f4e3595b7346921ea16c1f627f5"}
May 25 13:06:09.528447 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.528420    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql" event={"ID":"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8","Type":"ContainerStarted","Data":"7535c604fd133d6d81f68584f34449b62a0bd24c49dc30ed81f60d739a9b024f"}
May 25 13:06:09.529604 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.529580    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4" event={"ID":"127a81c2-9752-4c7f-979c-d5aa0d2a86b5","Type":"ContainerStarted","Data":"4c36e4e5d55852befd66656f959bfcaee81d29a6f006063b580cb0bb80f99ca9"}
May 25 13:06:09.532053 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.532027    2573 generic.go:358] "Generic (PLEG): container finished" podID="cda7f987-514d-47c2-b386-52b0eba3d858" containerID="828984ad35f40fc13eda9182a72cdc58516c2da7ed71d14f45182e12c365ac59" exitCode=0
May 25 13:06:09.532160 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.532094    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mkkfs" event={"ID":"cda7f987-514d-47c2-b386-52b0eba3d858","Type":"ContainerDied","Data":"828984ad35f40fc13eda9182a72cdc58516c2da7ed71d14f45182e12c365ac59"}
May 25 13:06:09.533474 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.533451    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wfc8p" event={"ID":"ab2520dc-3572-44d2-b557-7d15a0bc0c45","Type":"ContainerStarted","Data":"b2a6cc6d96cb187e483b2e0b6843fd28f2ce2fd1ea6e430dee1df3b2f81148ac"}
May 25 13:06:09.533574 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.533481    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wfc8p" event={"ID":"ab2520dc-3572-44d2-b557-7d15a0bc0c45","Type":"ContainerStarted","Data":"2f007d4a59192542f48a07e1b8c9e7c386bd0e5ffca8888b6c94b4d213fbb69f"}
May 25 13:06:09.534622 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.534585    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6" event={"ID":"05c02256-ea7d-4a74-8f08-a16384769838","Type":"ContainerStarted","Data":"61ca126f78c7e6cc124c48b5d6e7ac5f23ed3ef0e343cee048132c78d7677484"}
May 25 13:06:09.535577 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.535538    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr" event={"ID":"f3626d2c-129b-4a27-9855-4d9649bf3c0f","Type":"ContainerStarted","Data":"cb49d0ac73dedd570b3ea015b0f41ee17b751d9e68b2562a82ec6ae4dbac7b76"}
May 25 13:06:09.574528 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:09.574487    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-wfc8p" podStartSLOduration=2.574471264 podStartE2EDuration="2.574471264s" podCreationTimestamp="2026-05-25 13:06:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:06:09.573473615 +0000 UTC m=+36.835076446" watchObservedRunningTime="2026-05-25 13:06:09.574471264 +0000 UTC m=+36.836074097"
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:10.330667    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.330844    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:14.330824515 +0000 UTC m=+41.592427325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : configmap references non-existent config key: service-ca.crt
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:10.330957    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:10.331067    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:10.331107    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:10.331137    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.331247    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.331295    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.331309    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls podName:6e7fa479-726d-460c-b125-97d1adafa418 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:14.331289521 +0000 UTC m=+41.592892330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-j54sp" (UID: "6e7fa479-726d-460c-b125-97d1adafa418") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.331344    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls podName:5eda4f74-db07-4c34-af65-2136bf98fe2e nodeName:}" failed. No retries permitted until 2026-05-25 13:06:14.331323234 +0000 UTC m=+41.592926057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls") pod "cluster-samples-operator-697587599-q5h6p" (UID: "5eda4f74-db07-4c34-af65-2136bf98fe2e") : secret "samples-operator-tls" not found
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.331358    2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:10.331368 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.331382    2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb44d44c7-qzx96: secret "image-registry-tls" not found
May 25 13:06:10.332101 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.331449    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls podName:080f2207-b1de-4749-ab1d-a8e445abf8fa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:14.331430721 +0000 UTC m=+41.593033544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls") pod "image-registry-fb44d44c7-qzx96" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa") : secret "image-registry-tls" not found
May 25 13:06:10.332101 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.331367    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:10.332101 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.331497    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:14.331486917 +0000 UTC m=+41.593089758 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : secret "router-metrics-certs-default" not found
May 25 13:06:10.432501 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:10.431840    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:10.432501 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.432139    2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:06:10.432501 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.432196    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert podName:2f58bc38-d685-405e-aa7e-7de7a86001ee nodeName:}" failed. No retries permitted until 2026-05-25 13:06:14.432178317 +0000 UTC m=+41.693781130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert") pod "ingress-canary-hjzcb" (UID: "2f58bc38-d685-405e-aa7e-7de7a86001ee") : secret "canary-serving-cert" not found
May 25 13:06:10.532643 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:10.532514    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:10.533126 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.532771    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:06:10.533126 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:10.532840    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls podName:e0e32442-e801-4fcb-8c48-74b788e18965 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:14.532822059 +0000 UTC m=+41.794424869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls") pod "dns-default-q6pgt" (UID: "e0e32442-e801-4fcb-8c48-74b788e18965") : secret "dns-default-metrics-tls" not found
May 25 13:06:10.544840 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:10.544618    2573 generic.go:358] "Generic (PLEG): container finished" podID="cda7f987-514d-47c2-b386-52b0eba3d858" containerID="076d53ca430a0f717430a69808c14d641e1035ec01dcecee99f2c704226d4567" exitCode=0
May 25 13:06:10.544840 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:10.544686    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mkkfs" event={"ID":"cda7f987-514d-47c2-b386-52b0eba3d858","Type":"ContainerDied","Data":"076d53ca430a0f717430a69808c14d641e1035ec01dcecee99f2c704226d4567"}
May 25 13:06:14.366917 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:14.366878    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:14.366933    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:14.366965    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:14.367034    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.367062    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.367109    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.367127    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:22.36711299 +0000 UTC m=+49.628715804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : configmap references non-existent config key: service-ca.crt
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.367144    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls podName:6e7fa479-726d-460c-b125-97d1adafa418 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:22.367136463 +0000 UTC m=+49.628739272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-j54sp" (UID: "6e7fa479-726d-460c-b125-97d1adafa418") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:14.367167    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.367187    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.367231    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:22.367217425 +0000 UTC m=+49.628820249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : secret "router-metrics-certs-default" not found
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.367254    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls podName:5eda4f74-db07-4c34-af65-2136bf98fe2e nodeName:}" failed. No retries permitted until 2026-05-25 13:06:22.367245303 +0000 UTC m=+49.628848120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls") pod "cluster-samples-operator-697587599-q5h6p" (UID: "5eda4f74-db07-4c34-af65-2136bf98fe2e") : secret "samples-operator-tls" not found
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.367299    2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.367315    2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb44d44c7-qzx96: secret "image-registry-tls" not found
May 25 13:06:14.367477 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.367379    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls podName:080f2207-b1de-4749-ab1d-a8e445abf8fa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:22.367359819 +0000 UTC m=+49.628962646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls") pod "image-registry-fb44d44c7-qzx96" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa") : secret "image-registry-tls" not found
May 25 13:06:14.468171 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:14.468147    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:14.468341 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.468259    2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:06:14.468341 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.468317    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert podName:2f58bc38-d685-405e-aa7e-7de7a86001ee nodeName:}" failed. No retries permitted until 2026-05-25 13:06:22.468298444 +0000 UTC m=+49.729901257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert") pod "ingress-canary-hjzcb" (UID: "2f58bc38-d685-405e-aa7e-7de7a86001ee") : secret "canary-serving-cert" not found
May 25 13:06:14.569743 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:14.569714    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:14.569894 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.569857    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:06:14.569939 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:14.569913    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls podName:e0e32442-e801-4fcb-8c48-74b788e18965 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:22.569899303 +0000 UTC m=+49.831502112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls") pod "dns-default-q6pgt" (UID: "e0e32442-e801-4fcb-8c48-74b788e18965") : secret "dns-default-metrics-tls" not found
May 25 13:06:15.555504 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:15.555455    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sqk9d" event={"ID":"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59","Type":"ContainerStarted","Data":"5b8db30494900fb60308e5f04c1ce780161ab276107acc35f5d68fdf9d182a9f"}
May 25 13:06:15.559055 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:15.559026    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mkkfs" event={"ID":"cda7f987-514d-47c2-b386-52b0eba3d858","Type":"ContainerStarted","Data":"1144b8398910303c6686ed299479f4204fe71f0e0fde41f36a3fa8a9012c0bad"}
May 25 13:06:15.615077 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:15.613945    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mkkfs" podStartSLOduration=8.154400315 podStartE2EDuration="42.613927479s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="2026-05-25 13:05:34.563956041 +0000 UTC m=+1.825558889" lastFinishedPulling="2026-05-25 13:06:09.023483231 +0000 UTC m=+36.285086053" observedRunningTime="2026-05-25 13:06:15.612778702 +0000 UTC m=+42.874381535" watchObservedRunningTime="2026-05-25 13:06:15.613927479 +0000 UTC m=+42.875530312"
May 25 13:06:16.563439 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.563397    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6" event={"ID":"05c02256-ea7d-4a74-8f08-a16384769838","Type":"ContainerStarted","Data":"a922a07d482dec7d15b5d61b8447cc2f9c3f068aa090dd61334b3fec6c323395"}
May 25 13:06:16.565290 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.565260    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr" event={"ID":"f3626d2c-129b-4a27-9855-4d9649bf3c0f","Type":"ContainerStarted","Data":"8ff2dacb37885ed6663ea6bbef9457fc248e41a2abf5b299a558c23a856b09f1"}
May 25 13:06:16.567489 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.567469    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/0.log"
May 25 13:06:16.567605 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.567510    2573 generic.go:358] "Generic (PLEG): container finished" podID="ca0dc551-06fa-414c-a3e3-28f8d0b2e382" containerID="38a4c2615d4708b6cd80fc5937996cc242bdf2dd0720ebad680992ded02360e3" exitCode=255
May 25 13:06:16.567665 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.567623    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k" event={"ID":"ca0dc551-06fa-414c-a3e3-28f8d0b2e382","Type":"ContainerDied","Data":"38a4c2615d4708b6cd80fc5937996cc242bdf2dd0720ebad680992ded02360e3"}
May 25 13:06:16.567885 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.567868    2573 scope.go:117] "RemoveContainer" containerID="38a4c2615d4708b6cd80fc5937996cc242bdf2dd0720ebad680992ded02360e3"
May 25 13:06:16.569021 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.569000    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql" event={"ID":"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8","Type":"ContainerStarted","Data":"83fcdcfd0a3e7b817c87966170c766ac463f2c3f37f87aa9e822778ca794a738"}
May 25 13:06:16.578356 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.578307    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-ffjg6" podStartSLOduration=27.136497609 podStartE2EDuration="33.578293452s" podCreationTimestamp="2026-05-25 13:05:43 +0000 UTC" firstStartedPulling="2026-05-25 13:06:08.993784311 +0000 UTC m=+36.255387121" lastFinishedPulling="2026-05-25 13:06:15.435580141 +0000 UTC m=+42.697182964" observedRunningTime="2026-05-25 13:06:16.577778095 +0000 UTC m=+43.839380928" watchObservedRunningTime="2026-05-25 13:06:16.578293452 +0000 UTC m=+43.839896284"
May 25 13:06:16.580866 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.580842    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4" event={"ID":"127a81c2-9752-4c7f-979c-d5aa0d2a86b5","Type":"ContainerStarted","Data":"b7b872160de4a9760c0ee71c33dfe28775ce1e65dba9ef705d70afaa3bcfaf70"}
May 25 13:06:16.593439 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.593384    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dlmmr" podStartSLOduration=19.14003407 podStartE2EDuration="25.593368992s" podCreationTimestamp="2026-05-25 13:05:51 +0000 UTC" firstStartedPulling="2026-05-25 13:06:08.996336892 +0000 UTC m=+36.257939700" lastFinishedPulling="2026-05-25 13:06:15.449671813 +0000 UTC m=+42.711274622" observedRunningTime="2026-05-25 13:06:16.592736908 +0000 UTC m=+43.854339743" watchObservedRunningTime="2026-05-25 13:06:16.593368992 +0000 UTC m=+43.854971827"
May 25 13:06:16.612113 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.611930    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql" podStartSLOduration=26.169735787 podStartE2EDuration="32.611917926s" podCreationTimestamp="2026-05-25 13:05:44 +0000 UTC" firstStartedPulling="2026-05-25 13:06:08.993785269 +0000 UTC m=+36.255388081" lastFinishedPulling="2026-05-25 13:06:15.43596741 +0000 UTC m=+42.697570220" observedRunningTime="2026-05-25 13:06:16.611780552 +0000 UTC m=+43.873383386" watchObservedRunningTime="2026-05-25 13:06:16.611917926 +0000 UTC m=+43.873520758"
May 25 13:06:16.647408 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.647361    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-sqk9d" podStartSLOduration=35.205480989 podStartE2EDuration="41.647348414s" podCreationTimestamp="2026-05-25 13:05:35 +0000 UTC" firstStartedPulling="2026-05-25 13:06:08.99343563 +0000 UTC m=+36.255038451" lastFinishedPulling="2026-05-25 13:06:15.435303053 +0000 UTC m=+42.696905876" observedRunningTime="2026-05-25 13:06:16.647271561 +0000 UTC m=+43.908874391" watchObservedRunningTime="2026-05-25 13:06:16.647348414 +0000 UTC m=+43.908951247"
May 25 13:06:16.662369 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.662321    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4" podStartSLOduration=25.219717773 podStartE2EDuration="31.662306181s" podCreationTimestamp="2026-05-25 13:05:45 +0000 UTC" firstStartedPulling="2026-05-25 13:06:08.993396344 +0000 UTC m=+36.254999167" lastFinishedPulling="2026-05-25 13:06:15.435984744 +0000 UTC m=+42.697587575" observedRunningTime="2026-05-25 13:06:16.661769795 +0000 UTC m=+43.923372625" watchObservedRunningTime="2026-05-25 13:06:16.662306181 +0000 UTC m=+43.923909015"
May 25 13:06:16.995069 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.995043    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:16.995174 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:16.995073    2573 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:17.584197 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:17.584170    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:06:17.585815 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:17.584863    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/0.log"
May 25 13:06:17.585815 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:17.584902    2573 generic.go:358] "Generic (PLEG): container finished" podID="ca0dc551-06fa-414c-a3e3-28f8d0b2e382" containerID="f7a86de0708ee0bd7a4f8f8669c0ae548b8ce22f9eb398e73f09b25d7b75e582" exitCode=255
May 25 13:06:17.585815 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:17.585686    2573 scope.go:117] "RemoveContainer" containerID="f7a86de0708ee0bd7a4f8f8669c0ae548b8ce22f9eb398e73f09b25d7b75e582"
May 25 13:06:17.586043 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:17.585840    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k" event={"ID":"ca0dc551-06fa-414c-a3e3-28f8d0b2e382","Type":"ContainerDied","Data":"f7a86de0708ee0bd7a4f8f8669c0ae548b8ce22f9eb398e73f09b25d7b75e582"}
May 25 13:06:17.586043 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:17.585860    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-jwb2k_openshift-console-operator(ca0dc551-06fa-414c-a3e3-28f8d0b2e382)\"" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k" podUID="ca0dc551-06fa-414c-a3e3-28f8d0b2e382"
May 25 13:06:17.586043 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:17.585878    2573 scope.go:117] "RemoveContainer" containerID="38a4c2615d4708b6cd80fc5937996cc242bdf2dd0720ebad680992ded02360e3"
May 25 13:06:18.588901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:18.588875    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:06:18.589292 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:18.589175    2573 scope.go:117] "RemoveContainer" containerID="f7a86de0708ee0bd7a4f8f8669c0ae548b8ce22f9eb398e73f09b25d7b75e582"
May 25 13:06:18.589358 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:18.589340    2573 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-jwb2k_openshift-console-operator(ca0dc551-06fa-414c-a3e3-28f8d0b2e382)\"" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k" podUID="ca0dc551-06fa-414c-a3e3-28f8d0b2e382"
May 25 13:06:20.040724 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:20.040693    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wfc8p_ab2520dc-3572-44d2-b557-7d15a0bc0c45/dns-node-resolver/0.log"
May 25 13:06:20.842488 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:20.842461    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f78nn_880fa08b-8cfb-45ce-8c9c-b63dd8896b39/node-ca/0.log"
May 25 13:06:22.245313 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:22.245284    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-8kgql_a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8/kube-storage-version-migrator-operator/0.log"
May 25 13:06:22.438531 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:22.438499    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:22.438531 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:22.438534    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:22.438742 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:22.438660    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:22.438742 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.438679    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.438661609 +0000 UTC m=+65.700264418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : configmap references non-existent config key: service-ca.crt
May 25 13:06:22.438742 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.438723    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:22.438742 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:22.438731    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:22.438952 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:22.438770    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:22.438952 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.438784    2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:22.438952 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.438805    2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb44d44c7-qzx96: secret "image-registry-tls" not found
May 25 13:06:22.438952 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.438852    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:22.438952 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.438852    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:22.438952 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.438791    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls podName:6e7fa479-726d-460c-b125-97d1adafa418 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.438773081 +0000 UTC m=+65.700375890 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-j54sp" (UID: "6e7fa479-726d-460c-b125-97d1adafa418") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:22.438952 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.438940    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls podName:080f2207-b1de-4749-ab1d-a8e445abf8fa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.438924102 +0000 UTC m=+65.700526911 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls") pod "image-registry-fb44d44c7-qzx96" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa") : secret "image-registry-tls" not found
May 25 13:06:22.438952 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.438956    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls podName:5eda4f74-db07-4c34-af65-2136bf98fe2e nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.438947734 +0000 UTC m=+65.700550546 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls") pod "cluster-samples-operator-697587599-q5h6p" (UID: "5eda4f74-db07-4c34-af65-2136bf98fe2e") : secret "samples-operator-tls" not found
May 25 13:06:22.439225 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.438972    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs podName:21ef9931-fc75-4841-b015-3a450c700b77 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.438964488 +0000 UTC m=+65.700567299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs") pod "router-default-84964b69b-gj5s2" (UID: "21ef9931-fc75-4841-b015-3a450c700b77") : secret "router-metrics-certs-default" not found
May 25 13:06:22.540323 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:22.540260    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:22.540428 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.540406    2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:06:22.540482 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.540473    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert podName:2f58bc38-d685-405e-aa7e-7de7a86001ee nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.54045882 +0000 UTC m=+65.802061629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert") pod "ingress-canary-hjzcb" (UID: "2f58bc38-d685-405e-aa7e-7de7a86001ee") : secret "canary-serving-cert" not found
May 25 13:06:22.641436 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:22.641411    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:22.641562 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.641529    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:06:22.641611 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:06:22.641589    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls podName:e0e32442-e801-4fcb-8c48-74b788e18965 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.641578396 +0000 UTC m=+65.903181205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls") pod "dns-default-q6pgt" (UID: "e0e32442-e801-4fcb-8c48-74b788e18965") : secret "dns-default-metrics-tls" not found
May 25 13:06:26.994703 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:26.994668    2573 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:26.994703 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:26.994707    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:26.995202 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:26.995057    2573 scope.go:117] "RemoveContainer" containerID="f7a86de0708ee0bd7a4f8f8669c0ae548b8ce22f9eb398e73f09b25d7b75e582"
May 25 13:06:27.613092 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:27.613065    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:06:27.613255 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:27.613132    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k" event={"ID":"ca0dc551-06fa-414c-a3e3-28f8d0b2e382","Type":"ContainerStarted","Data":"2383063b1f1a85130140c964d02bb76fc8ffccbca0ffe53b356387d90ccdade1"}
May 25 13:06:27.613417 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:27.613396    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:27.618082 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:27.618061    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k"
May 25 13:06:27.633789 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:27.633747    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-jwb2k" podStartSLOduration=38.203541853 podStartE2EDuration="44.633735426s" podCreationTimestamp="2026-05-25 13:05:43 +0000 UTC" firstStartedPulling="2026-05-25 13:06:09.00538412 +0000 UTC m=+36.266986933" lastFinishedPulling="2026-05-25 13:06:15.435577696 +0000 UTC m=+42.697180506" observedRunningTime="2026-05-25 13:06:27.632865185 +0000 UTC m=+54.894468017" watchObservedRunningTime="2026-05-25 13:06:27.633735426 +0000 UTC m=+54.895338257"
May 25 13:06:32.532060 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:32.532027    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-684s7"
May 25 13:06:38.161702 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.161666    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:06:38.162146 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.161781    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:06:38.164421 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.164398    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 13:06:38.164573 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.164470    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 13:06:38.175910 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.175887    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430200e3-29df-46e6-99ce-346a015400ae-original-pull-secret\") pod \"global-pull-secret-syncer-92l8w\" (UID: \"430200e3-29df-46e6-99ce-346a015400ae\") " pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:06:38.176012 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.175912    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1774200-9350-464d-a3b8-2be8812bcde4-metrics-certs\") pod \"network-metrics-daemon-tkljt\" (UID: \"c1774200-9350-464d-a3b8-2be8812bcde4\") " pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:06:38.249439 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.249414    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-clhmx\""
May 25 13:06:38.258340 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.258321    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkljt"
May 25 13:06:38.259893 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.259870    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l8w"
May 25 13:06:38.262494 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.262478    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6sj\" (UniqueName: \"kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj\") pod \"network-check-target-7xhlm\" (UID: \"bd26de80-992a-449c-b62d-608a45a4fa12\") " pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:06:38.264814 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.264784    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m6sj\" (UniqueName: \"kubernetes.io/projected/bd26de80-992a-449c-b62d-608a45a4fa12-kube-api-access-4m6sj\") pod \"network-check-target-7xhlm\" (UID: \"bd26de80-992a-449c-b62d-608a45a4fa12\") " pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:06:38.385191 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.385163    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tkljt"]
May 25 13:06:38.389495 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:38.389457    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1774200_9350_464d_a3b8_2be8812bcde4.slice/crio-6cc752ee2f3e5d5224b238a39dca6fc55e4886af5cca89ca53be26d5aa8160d2 WatchSource:0}: Error finding container 6cc752ee2f3e5d5224b238a39dca6fc55e4886af5cca89ca53be26d5aa8160d2: Status 404 returned error can't find the container with id 6cc752ee2f3e5d5224b238a39dca6fc55e4886af5cca89ca53be26d5aa8160d2
May 25 13:06:38.400997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.400971    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-92l8w"]
May 25 13:06:38.403291 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:38.403271    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430200e3_29df_46e6_99ce_346a015400ae.slice/crio-d09e275cc0c3e3a409be8bc6394934267939cb05a31d459f2a2a3695255c6a16 WatchSource:0}: Error finding container d09e275cc0c3e3a409be8bc6394934267939cb05a31d459f2a2a3695255c6a16: Status 404 returned error can't find the container with id d09e275cc0c3e3a409be8bc6394934267939cb05a31d459f2a2a3695255c6a16
May 25 13:06:38.463719 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.463667    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:38.463719 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.463709    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:38.463841 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.463782    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:38.463841 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.463809    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:38.463923 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.463841    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:38.464370 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.464323    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ef9931-fc75-4841-b015-3a450c700b77-service-ca-bundle\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:38.466091 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.466072    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e7fa479-726d-460c-b125-97d1adafa418-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-j54sp\" (UID: \"6e7fa479-726d-460c-b125-97d1adafa418\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:38.466174 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.466076    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ef9931-fc75-4841-b015-3a450c700b77-metrics-certs\") pod \"router-default-84964b69b-gj5s2\" (UID: \"21ef9931-fc75-4841-b015-3a450c700b77\") " pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:38.466434 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.466418    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") pod \"image-registry-fb44d44c7-qzx96\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") " pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:38.466508 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.466481    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eda4f74-db07-4c34-af65-2136bf98fe2e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-q5h6p\" (UID: \"5eda4f74-db07-4c34-af65-2136bf98fe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:38.556807 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.556775    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcddr\""
May 25 13:06:38.564739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.564722    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:38.564851 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.564832    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:06:38.566696 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.566678    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f58bc38-d685-405e-aa7e-7de7a86001ee-cert\") pod \"ingress-canary-hjzcb\" (UID: \"2f58bc38-d685-405e-aa7e-7de7a86001ee\") " pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:38.644851 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.644814    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-92l8w" event={"ID":"430200e3-29df-46e6-99ce-346a015400ae","Type":"ContainerStarted","Data":"d09e275cc0c3e3a409be8bc6394934267939cb05a31d459f2a2a3695255c6a16"}
May 25 13:06:38.645972 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.645948    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tkljt" event={"ID":"c1774200-9350-464d-a3b8-2be8812bcde4","Type":"ContainerStarted","Data":"6cc752ee2f3e5d5224b238a39dca6fc55e4886af5cca89ca53be26d5aa8160d2"}
May 25 13:06:38.665472 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.665448    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:38.667501 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.667481    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e32442-e801-4fcb-8c48-74b788e18965-metrics-tls\") pod \"dns-default-q6pgt\" (UID: \"e0e32442-e801-4fcb-8c48-74b788e18965\") " pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:38.667849 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.667831    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-96p87\""
May 25 13:06:38.675422 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.675399    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"
May 25 13:06:38.677751 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.677695    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lntzn\""
May 25 13:06:38.678742 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.678724    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7xhlm"]
May 25 13:06:38.682011 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:38.681990    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd26de80_992a_449c_b62d_608a45a4fa12.slice/crio-074bb48d53f06a022b1cf8b4ef55629662e45075f395f7f5a8562d848b6c07ec WatchSource:0}: Error finding container 074bb48d53f06a022b1cf8b4ef55629662e45075f395f7f5a8562d848b6c07ec: Status 404 returned error can't find the container with id 074bb48d53f06a022b1cf8b4ef55629662e45075f395f7f5a8562d848b6c07ec
May 25 13:06:38.685707 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.685688    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:38.712477 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.712454    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6ch8m\""
May 25 13:06:38.720828 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.720768    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:38.728933 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.728878    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fz5sg\""
May 25 13:06:38.739352 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.738969    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"
May 25 13:06:38.828696 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.828666    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp"]
May 25 13:06:38.833048 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:38.833011    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7fa479_726d_460c_b125_97d1adafa418.slice/crio-e857ddba08901bdc6461e39a55b4b02c73d07a36f2104eb1099ef030d9fc2b93 WatchSource:0}: Error finding container e857ddba08901bdc6461e39a55b4b02c73d07a36f2104eb1099ef030d9fc2b93: Status 404 returned error can't find the container with id e857ddba08901bdc6461e39a55b4b02c73d07a36f2104eb1099ef030d9fc2b93
May 25 13:06:38.842100 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.842054    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vbzlk\""
May 25 13:06:38.844880 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.844856    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb44d44c7-qzx96"]
May 25 13:06:38.848261 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:38.848238    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080f2207_b1de_4749_ab1d_a8e445abf8fa.slice/crio-9b81af125950005fef0d36cd25dda313be7ef851f6c8de77a19567ceed3711e1 WatchSource:0}: Error finding container 9b81af125950005fef0d36cd25dda313be7ef851f6c8de77a19567ceed3711e1: Status 404 returned error can't find the container with id 9b81af125950005fef0d36cd25dda313be7ef851f6c8de77a19567ceed3711e1
May 25 13:06:38.850422 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.849936    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjzcb"
May 25 13:06:38.853752 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.853490    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5d4pm\""
May 25 13:06:38.861361 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.861341    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:38.888177 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.888073    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84964b69b-gj5s2"]
May 25 13:06:38.890370 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:38.890333    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ef9931_fc75_4841_b015_3a450c700b77.slice/crio-9787cd51ff6518e437815e3d46a862600b47d53aebfd3892f420f008383562ea WatchSource:0}: Error finding container 9787cd51ff6518e437815e3d46a862600b47d53aebfd3892f420f008383562ea: Status 404 returned error can't find the container with id 9787cd51ff6518e437815e3d46a862600b47d53aebfd3892f420f008383562ea
May 25 13:06:38.910451 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:38.910085    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p"]
May 25 13:06:39.025670 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.025622    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjzcb"]
May 25 13:06:39.029307 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:39.029260    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f58bc38_d685_405e_aa7e_7de7a86001ee.slice/crio-092864d55220edd68e35035014e3d4571f7b48a13586d5af9e414f73a260b4b7 WatchSource:0}: Error finding container 092864d55220edd68e35035014e3d4571f7b48a13586d5af9e414f73a260b4b7: Status 404 returned error can't find the container with id 092864d55220edd68e35035014e3d4571f7b48a13586d5af9e414f73a260b4b7
May 25 13:06:39.055594 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.055533    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q6pgt"]
May 25 13:06:39.059337 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:39.059307    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e32442_e801_4fcb_8c48_74b788e18965.slice/crio-21f2e692cefc7ccb930e8696718074f28abf0c0d3da1bef7ffb8e2211d7d3c44 WatchSource:0}: Error finding container 21f2e692cefc7ccb930e8696718074f28abf0c0d3da1bef7ffb8e2211d7d3c44: Status 404 returned error can't find the container with id 21f2e692cefc7ccb930e8696718074f28abf0c0d3da1bef7ffb8e2211d7d3c44
May 25 13:06:39.653829 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.653787    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p" event={"ID":"5eda4f74-db07-4c34-af65-2136bf98fe2e","Type":"ContainerStarted","Data":"a4a25d8c89e56823f5bf294a443155a992b5363c24c06e7f44c586490d6c420c"}
May 25 13:06:39.658470 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.658437    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84964b69b-gj5s2" event={"ID":"21ef9931-fc75-4841-b015-3a450c700b77","Type":"ContainerStarted","Data":"912cf8b448d5b4ca44595590fa5709c13b4b0fdf99c0df35c2e8425f02331382"}
May 25 13:06:39.658470 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.658477    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84964b69b-gj5s2" event={"ID":"21ef9931-fc75-4841-b015-3a450c700b77","Type":"ContainerStarted","Data":"9787cd51ff6518e437815e3d46a862600b47d53aebfd3892f420f008383562ea"}
May 25 13:06:39.661212 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.661115    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7xhlm" event={"ID":"bd26de80-992a-449c-b62d-608a45a4fa12","Type":"ContainerStarted","Data":"5d5f49b9321919754b1876b99dee509eac7dec21bc583eacb490290d4d8a02c6"}
May 25 13:06:39.661212 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.661148    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7xhlm" event={"ID":"bd26de80-992a-449c-b62d-608a45a4fa12","Type":"ContainerStarted","Data":"074bb48d53f06a022b1cf8b4ef55629662e45075f395f7f5a8562d848b6c07ec"}
May 25 13:06:39.661212 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.661193    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:06:39.664107 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.664062    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjzcb" event={"ID":"2f58bc38-d685-405e-aa7e-7de7a86001ee","Type":"ContainerStarted","Data":"092864d55220edd68e35035014e3d4571f7b48a13586d5af9e414f73a260b4b7"}
May 25 13:06:39.667035 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.667011    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96" event={"ID":"080f2207-b1de-4749-ab1d-a8e445abf8fa","Type":"ContainerStarted","Data":"2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1"}
May 25 13:06:39.667137 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.667044    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96" event={"ID":"080f2207-b1de-4749-ab1d-a8e445abf8fa","Type":"ContainerStarted","Data":"9b81af125950005fef0d36cd25dda313be7ef851f6c8de77a19567ceed3711e1"}
May 25 13:06:39.667747 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.667627    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:06:39.669445 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.669422    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp" event={"ID":"6e7fa479-726d-460c-b125-97d1adafa418","Type":"ContainerStarted","Data":"e857ddba08901bdc6461e39a55b4b02c73d07a36f2104eb1099ef030d9fc2b93"}
May 25 13:06:39.670576 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.670539    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q6pgt" event={"ID":"e0e32442-e801-4fcb-8c48-74b788e18965","Type":"ContainerStarted","Data":"21f2e692cefc7ccb930e8696718074f28abf0c0d3da1bef7ffb8e2211d7d3c44"}
May 25 13:06:39.721927 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.721900    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:39.725119 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.725083    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:39.728360 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.727142    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-84964b69b-gj5s2" podStartSLOduration=64.727125709 podStartE2EDuration="1m4.727125709s" podCreationTimestamp="2026-05-25 13:05:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:06:39.693520194 +0000 UTC m=+66.955123026" watchObservedRunningTime="2026-05-25 13:06:39.727125709 +0000 UTC m=+66.988728543"
May 25 13:06:39.752788 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.752744    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96" podStartSLOduration=66.752728678 podStartE2EDuration="1m6.752728678s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:06:39.728247643 +0000 UTC m=+66.989850515" watchObservedRunningTime="2026-05-25 13:06:39.752728678 +0000 UTC m=+67.014331511"
May 25 13:06:39.754264 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:39.753341    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7xhlm" podStartSLOduration=66.753330218 podStartE2EDuration="1m6.753330218s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:06:39.753197456 +0000 UTC m=+67.014800301" watchObservedRunningTime="2026-05-25 13:06:39.753330218 +0000 UTC m=+67.014933108"
May 25 13:06:40.680240 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:40.679046    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:40.680710 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:40.680457    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-84964b69b-gj5s2"
May 25 13:06:41.079133 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.079063    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"]
May 25 13:06:41.098900 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.098862    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hlx8q"]
May 25 13:06:41.099187 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.099169    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.102715 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.102444    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 25 13:06:41.102715 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.102486    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 25 13:06:41.102715 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.102663    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 25 13:06:41.105125 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.104070    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 25 13:06:41.105125 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.104338    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 25 13:06:41.105125 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.104602    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 25 13:06:41.105125 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.104886    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 25 13:06:41.128401 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.128148    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"]
May 25 13:06:41.128401 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.128184    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hlx8q"]
May 25 13:06:41.128401 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.128311    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.131273 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.131100    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wgqp9\""
May 25 13:06:41.131963 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.131500    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 13:06:41.131963 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.131802    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 13:06:41.186918 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.186876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.187091 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.186942    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-crio-socket\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.187091 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.187017    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-ca\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.187091 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.187044    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/7213f920-4553-4f25-bd78-99b4997d22f4-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.187091 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.187077    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-hub\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.187427 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.187093    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzqbq\" (UniqueName: \"kubernetes.io/projected/7213f920-4553-4f25-bd78-99b4997d22f4-kube-api-access-fzqbq\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.187427 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.187116    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-data-volume\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.187427 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.187144    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.187427 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.187160    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.187427 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.187179    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.187427 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.187230    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vz7\" (UniqueName: \"kubernetes.io/projected/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-kube-api-access-b4vz7\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.288090 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288054    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.288263 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288123    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-crio-socket\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.288263 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288155    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-ca\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.288263 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288181    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/7213f920-4553-4f25-bd78-99b4997d22f4-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.288263 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288216    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-hub\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.288263 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288241    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fzqbq\" (UniqueName: \"kubernetes.io/projected/7213f920-4553-4f25-bd78-99b4997d22f4-kube-api-access-fzqbq\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.288488 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288268    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-data-volume\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.288488 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288302    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.288488 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288326    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.288488 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288351    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.288488 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288386    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vz7\" (UniqueName: \"kubernetes.io/projected/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-kube-api-access-b4vz7\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.288745 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288700    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-crio-socket\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.288808 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.288741    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.289366 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.289210    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-data-volume\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.289706 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.289661    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/7213f920-4553-4f25-bd78-99b4997d22f4-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.291821 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.291766    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-ca\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.292103 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.292064    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.293684 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.293662    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-hub\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.293778 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.293687    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7213f920-4553-4f25-bd78-99b4997d22f4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.294176 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.294134    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.320939 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.320914    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vz7\" (UniqueName: \"kubernetes.io/projected/c2367bd3-2167-4ca1-be1c-01c25e1bb1d7-kube-api-access-b4vz7\") pod \"insights-runtime-extractor-hlx8q\" (UID: \"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7\") " pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:41.321288 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.321240    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzqbq\" (UniqueName: \"kubernetes.io/projected/7213f920-4553-4f25-bd78-99b4997d22f4-kube-api-access-fzqbq\") pod \"cluster-proxy-proxy-agent-6bbb5c9bf-pvphf\" (UID: \"7213f920-4553-4f25-bd78-99b4997d22f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.425753 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.425678    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"
May 25 13:06:41.443409 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:41.443387    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hlx8q"
May 25 13:06:44.875939 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:44.875907    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hlx8q"]
May 25 13:06:44.902739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:44.902711    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf"]
May 25 13:06:44.908469 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:44.908042    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7213f920_4553_4f25_bd78_99b4997d22f4.slice/crio-13de943299ec3dd4baca157ebed96bd2dabbab84182cda2b36331e351972d0ce WatchSource:0}: Error finding container 13de943299ec3dd4baca157ebed96bd2dabbab84182cda2b36331e351972d0ce: Status 404 returned error can't find the container with id 13de943299ec3dd4baca157ebed96bd2dabbab84182cda2b36331e351972d0ce
May 25 13:06:45.694692 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.694612    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp" event={"ID":"6e7fa479-726d-460c-b125-97d1adafa418","Type":"ContainerStarted","Data":"3d4960f2b96e39529a40df5b978d372422b346cf2f34f1b682f37993b86278fd"}
May 25 13:06:45.698174 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.698147    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q6pgt" event={"ID":"e0e32442-e801-4fcb-8c48-74b788e18965","Type":"ContainerStarted","Data":"2b199c3593106650c6fea108251f402f4f2cc7829cb25b9fc4a0bcd8b7cd4e7a"}
May 25 13:06:45.698300 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.698186    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:45.698300 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.698202    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q6pgt" event={"ID":"e0e32442-e801-4fcb-8c48-74b788e18965","Type":"ContainerStarted","Data":"6ae6b234d756e887f4a6094bc2859ca4e41eaac57dedc335004a7f5f4c56e899"}
May 25 13:06:45.701452 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.701391    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p" event={"ID":"5eda4f74-db07-4c34-af65-2136bf98fe2e","Type":"ContainerStarted","Data":"2b54430d80a2bb8c9b0230aca8d811f7853108920f65448b8f0734317357489c"}
May 25 13:06:45.701452 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.701423    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p" event={"ID":"5eda4f74-db07-4c34-af65-2136bf98fe2e","Type":"ContainerStarted","Data":"b2abf641a14ff54c78f0206771f4e5abfaa994dbc3ca1076be20b152f8a8bc3d"}
May 25 13:06:45.703043 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.702997    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlx8q" event={"ID":"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7","Type":"ContainerStarted","Data":"04d7f35fb1c8fde850ba83c820384288214af3eac7f157989c0dc47c0727559a"}
May 25 13:06:45.703043 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.703028    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlx8q" event={"ID":"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7","Type":"ContainerStarted","Data":"e72a307adbe8760bb90cb3f883c39b88b074aa170692c035305d3eae42b5276c"}
May 25 13:06:45.704990 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.704967    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf" event={"ID":"7213f920-4553-4f25-bd78-99b4997d22f4","Type":"ContainerStarted","Data":"13de943299ec3dd4baca157ebed96bd2dabbab84182cda2b36331e351972d0ce"}
May 25 13:06:45.708140 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.708115    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-92l8w" event={"ID":"430200e3-29df-46e6-99ce-346a015400ae","Type":"ContainerStarted","Data":"9385d31f180d7b216022b0416f5448ec9bf0a43ab32c016e26a76b1414a8ecbb"}
May 25 13:06:45.710938 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.710770    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tkljt" event={"ID":"c1774200-9350-464d-a3b8-2be8812bcde4","Type":"ContainerStarted","Data":"6ad56a64e8e4d7d08f795d7fbd6bf2a97adfa68df425d1b8652a490b9b7d3af6"}
May 25 13:06:45.710938 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.710798    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tkljt" event={"ID":"c1774200-9350-464d-a3b8-2be8812bcde4","Type":"ContainerStarted","Data":"5d5d5ae4e1f2578468653daff20dc2a028552e6deb23fe2cc1f4a81ed9701704"}
May 25 13:06:45.720995 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.720285    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-j54sp" podStartSLOduration=65.826822087 podStartE2EDuration="1m11.720270413s" podCreationTimestamp="2026-05-25 13:05:34 +0000 UTC" firstStartedPulling="2026-05-25 13:06:38.834798989 +0000 UTC m=+66.096401812" lastFinishedPulling="2026-05-25 13:06:44.728247314 +0000 UTC m=+71.989850138" observedRunningTime="2026-05-25 13:06:45.718267568 +0000 UTC m=+72.979870403" watchObservedRunningTime="2026-05-25 13:06:45.720270413 +0000 UTC m=+72.981873245"
May 25 13:06:45.720995 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.720353    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjzcb" event={"ID":"2f58bc38-d685-405e-aa7e-7de7a86001ee","Type":"ContainerStarted","Data":"adc9daa4a9b1acf06bca2add7adec6a29baf5bed7b614ce4c85cbe26959db9b5"}
May 25 13:06:45.750966 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.750917    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tkljt" podStartSLOduration=66.960787693 podStartE2EDuration="1m12.750898931s" podCreationTimestamp="2026-05-25 13:05:33 +0000 UTC" firstStartedPulling="2026-05-25 13:06:38.391467459 +0000 UTC m=+65.653070268" lastFinishedPulling="2026-05-25 13:06:44.181578684 +0000 UTC m=+71.443181506" observedRunningTime="2026-05-25 13:06:45.750507675 +0000 UTC m=+73.012110508" watchObservedRunningTime="2026-05-25 13:06:45.750898931 +0000 UTC m=+73.012501763"
May 25 13:06:45.779098 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.779060    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-q5h6p" podStartSLOduration=58.054454239 podStartE2EDuration="1m3.779045512s" podCreationTimestamp="2026-05-25 13:05:42 +0000 UTC" firstStartedPulling="2026-05-25 13:06:39.005229633 +0000 UTC m=+66.266832456" lastFinishedPulling="2026-05-25 13:06:44.72982092 +0000 UTC m=+71.991423729" observedRunningTime="2026-05-25 13:06:45.777348905 +0000 UTC m=+73.038951736" watchObservedRunningTime="2026-05-25 13:06:45.779045512 +0000 UTC m=+73.040648344"
May 25 13:06:45.804265 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.804220    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-92l8w" podStartSLOduration=65.479300997 podStartE2EDuration="1m11.80420538s" podCreationTimestamp="2026-05-25 13:05:34 +0000 UTC" firstStartedPulling="2026-05-25 13:06:38.404886653 +0000 UTC m=+65.666489462" lastFinishedPulling="2026-05-25 13:06:44.729791021 +0000 UTC m=+71.991393845" observedRunningTime="2026-05-25 13:06:45.803192394 +0000 UTC m=+73.064795226" watchObservedRunningTime="2026-05-25 13:06:45.80420538 +0000 UTC m=+73.065808212"
May 25 13:06:45.827836 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.827676    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-q6pgt" podStartSLOduration=34.159896904 podStartE2EDuration="39.827665879s" podCreationTimestamp="2026-05-25 13:06:06 +0000 UTC" firstStartedPulling="2026-05-25 13:06:39.061839783 +0000 UTC m=+66.323442596" lastFinishedPulling="2026-05-25 13:06:44.729608744 +0000 UTC m=+71.991211571" observedRunningTime="2026-05-25 13:06:45.826013123 +0000 UTC m=+73.087615968" watchObservedRunningTime="2026-05-25 13:06:45.827665879 +0000 UTC m=+73.089268710"
May 25 13:06:45.857755 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:45.857694    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hjzcb" podStartSLOduration=33.773020302 podStartE2EDuration="39.857656374s" podCreationTimestamp="2026-05-25 13:06:06 +0000 UTC" firstStartedPulling="2026-05-25 13:06:39.032572962 +0000 UTC m=+66.294175774" lastFinishedPulling="2026-05-25 13:06:45.117209031 +0000 UTC m=+72.378811846" observedRunningTime="2026-05-25 13:06:45.852202581 +0000 UTC m=+73.113805413" watchObservedRunningTime="2026-05-25 13:06:45.857656374 +0000 UTC m=+73.119259208"
May 25 13:06:48.734637 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:48.734574    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlx8q" event={"ID":"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7","Type":"ContainerStarted","Data":"d7721f5d229b32826184b57a01456069bfbd8baa8265813c6abf3359092c045a"}
May 25 13:06:48.736485 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:48.736431    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf" event={"ID":"7213f920-4553-4f25-bd78-99b4997d22f4","Type":"ContainerStarted","Data":"9ddebb44f9506abed879bf1fb2d283d21907754330b58e0d0bfa52a7281af006"}
May 25 13:06:50.744300 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:50.744266    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlx8q" event={"ID":"c2367bd3-2167-4ca1-be1c-01c25e1bb1d7","Type":"ContainerStarted","Data":"b9344d5aa40bd6edeae7317d32b4c17ed3d9f534f44d46f81c65759062453fe8"}
May 25 13:06:50.746128 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:50.746104    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf" event={"ID":"7213f920-4553-4f25-bd78-99b4997d22f4","Type":"ContainerStarted","Data":"00312223681c784f863a614af82a558b246a04631fb55306c08c95a571e9d016"}
May 25 13:06:50.746235 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:50.746138    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf" event={"ID":"7213f920-4553-4f25-bd78-99b4997d22f4","Type":"ContainerStarted","Data":"22425b1a091fd557190f0197b9b9a8306614f8358dc9b1189d264f4fff4634e6"}
May 25 13:06:50.763923 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:50.763878    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hlx8q" podStartSLOduration=4.453335847 podStartE2EDuration="9.763865903s" podCreationTimestamp="2026-05-25 13:06:41 +0000 UTC" firstStartedPulling="2026-05-25 13:06:45.207780619 +0000 UTC m=+72.469383432" lastFinishedPulling="2026-05-25 13:06:50.518310678 +0000 UTC m=+77.779913488" observedRunningTime="2026-05-25 13:06:50.762575801 +0000 UTC m=+78.024178632" watchObservedRunningTime="2026-05-25 13:06:50.763865903 +0000 UTC m=+78.025468724"
May 25 13:06:50.786863 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:50.786787    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbb5c9bf-pvphf" podStartSLOduration=4.149289053 podStartE2EDuration="9.786775065s" podCreationTimestamp="2026-05-25 13:06:41 +0000 UTC" firstStartedPulling="2026-05-25 13:06:44.910748524 +0000 UTC m=+72.172351349" lastFinishedPulling="2026-05-25 13:06:50.548234552 +0000 UTC m=+77.809837361" observedRunningTime="2026-05-25 13:06:50.785703616 +0000 UTC m=+78.047306448" watchObservedRunningTime="2026-05-25 13:06:50.786775065 +0000 UTC m=+78.048377896"
May 25 13:06:54.923117 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:54.923082    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gp65l"]
May 25 13:06:54.926217 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:54.926192    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:54.928716 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:54.928694    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 13:06:54.928853 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:54.928797    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 13:06:54.928853 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:54.928797    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-47s74\""
May 25 13:06:54.929594 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:54.929579    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 13:06:54.929677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:54.929595    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 13:06:55.007069 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.007046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-root\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.007173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.007080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-textfile\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.007173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.007125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-wtmp\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.007173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.007143    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-tls\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.007173 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.007164    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82plg\" (UniqueName: \"kubernetes.io/projected/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-kube-api-access-82plg\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.007298 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.007184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-metrics-client-ca\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.007298 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.007229    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-sys\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.007298 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.007264    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.007387 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.007312    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108076 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108044    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-sys\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108076 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108076    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108249 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108109    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108249 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108137    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-root\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108249 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108155    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-textfile\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108249 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108161    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-sys\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108249 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108172    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-wtmp\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108249 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108238    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-tls\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108253    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-root\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108275    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-wtmp\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108276    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82plg\" (UniqueName: \"kubernetes.io/projected/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-kube-api-access-82plg\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108498 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108327    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-metrics-client-ca\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108724 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108678    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-textfile\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108767 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108734    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.108860 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.108844    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-metrics-client-ca\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.110622 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.110598    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.110687 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.110672    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-node-exporter-tls\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.116051 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.115996    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82plg\" (UniqueName: \"kubernetes.io/projected/01c0855a-d37c-4cd0-bb3a-2713d13f0bc3-kube-api-access-82plg\") pod \"node-exporter-gp65l\" (UID: \"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3\") " pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.235334 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.235269    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gp65l"
May 25 13:06:55.244132 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:06:55.244092    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c0855a_d37c_4cd0_bb3a_2713d13f0bc3.slice/crio-eb0bb7e750aedadd5007e25613ae4dc0a048829aec9159b9cf4a8fcdb5840f0c WatchSource:0}: Error finding container eb0bb7e750aedadd5007e25613ae4dc0a048829aec9159b9cf4a8fcdb5840f0c: Status 404 returned error can't find the container with id eb0bb7e750aedadd5007e25613ae4dc0a048829aec9159b9cf4a8fcdb5840f0c
May 25 13:06:55.729357 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.729324    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q6pgt"
May 25 13:06:55.762249 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:55.762209    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gp65l" event={"ID":"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3","Type":"ContainerStarted","Data":"eb0bb7e750aedadd5007e25613ae4dc0a048829aec9159b9cf4a8fcdb5840f0c"}
May 25 13:06:56.766299 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:56.766266    2573 generic.go:358] "Generic (PLEG): container finished" podID="01c0855a-d37c-4cd0-bb3a-2713d13f0bc3" containerID="bc75dc4281a9446a3f56564d99e055797cf27d6493cdc71bf0413f12c4924482" exitCode=0
May 25 13:06:56.766678 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:56.766319    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gp65l" event={"ID":"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3","Type":"ContainerDied","Data":"bc75dc4281a9446a3f56564d99e055797cf27d6493cdc71bf0413f12c4924482"}
May 25 13:06:57.770890 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:57.770858    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gp65l" event={"ID":"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3","Type":"ContainerStarted","Data":"70c604265666359e945719353aa12dc52e9894ed4a8298708a36b140f9bd2d7d"}
May 25 13:06:57.770890 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:57.770893    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gp65l" event={"ID":"01c0855a-d37c-4cd0-bb3a-2713d13f0bc3","Type":"ContainerStarted","Data":"4250271cb589414495c3ebf1c08bdc1802e5ad599f6430f258c7f6329f437eda"}
May 25 13:06:57.789689 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:57.789645    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gp65l" podStartSLOduration=2.986177092 podStartE2EDuration="3.789630042s" podCreationTimestamp="2026-05-25 13:06:54 +0000 UTC" firstStartedPulling="2026-05-25 13:06:55.246066168 +0000 UTC m=+82.507668985" lastFinishedPulling="2026-05-25 13:06:56.049519117 +0000 UTC m=+83.311121935" observedRunningTime="2026-05-25 13:06:57.788299326 +0000 UTC m=+85.049902156" watchObservedRunningTime="2026-05-25 13:06:57.789630042 +0000 UTC m=+85.051232874"
May 25 13:06:58.689687 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:58.689656    2573 patch_prober.go:28] interesting pod/image-registry-fb44d44c7-qzx96 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 13:06:58.689815 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:06:58.689705    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96" podUID="080f2207-b1de-4749-ab1d-a8e445abf8fa" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:07:01.684778 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:01.684751    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:07:05.304099 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:05.304062    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb44d44c7-qzx96"]
May 25 13:07:10.680990 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:10.680960    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7xhlm"
May 25 13:07:21.836224 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:21.836196    2573 generic.go:358] "Generic (PLEG): container finished" podID="a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59" containerID="5b8db30494900fb60308e5f04c1ce780161ab276107acc35f5d68fdf9d182a9f" exitCode=0
May 25 13:07:21.836594 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:21.836264    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sqk9d" event={"ID":"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59","Type":"ContainerDied","Data":"5b8db30494900fb60308e5f04c1ce780161ab276107acc35f5d68fdf9d182a9f"}
May 25 13:07:21.836668 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:21.836624    2573 scope.go:117] "RemoveContainer" containerID="5b8db30494900fb60308e5f04c1ce780161ab276107acc35f5d68fdf9d182a9f"
May 25 13:07:21.837723 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:21.837697    2573 generic.go:358] "Generic (PLEG): container finished" podID="127a81c2-9752-4c7f-979c-d5aa0d2a86b5" containerID="b7b872160de4a9760c0ee71c33dfe28775ce1e65dba9ef705d70afaa3bcfaf70" exitCode=0
May 25 13:07:21.837829 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:21.837768    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4" event={"ID":"127a81c2-9752-4c7f-979c-d5aa0d2a86b5","Type":"ContainerDied","Data":"b7b872160de4a9760c0ee71c33dfe28775ce1e65dba9ef705d70afaa3bcfaf70"}
May 25 13:07:21.838135 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:21.838036    2573 scope.go:117] "RemoveContainer" containerID="b7b872160de4a9760c0ee71c33dfe28775ce1e65dba9ef705d70afaa3bcfaf70"
May 25 13:07:22.287053 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:22.287021    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84964b69b-gj5s2_21ef9931-fc75-4841-b015-3a450c700b77/router/0.log"
May 25 13:07:22.298370 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:22.298347    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjzcb_2f58bc38-d685-405e-aa7e-7de7a86001ee/serve-healthcheck-canary/0.log"
May 25 13:07:22.841971 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:22.841937    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sqk9d" event={"ID":"a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59","Type":"ContainerStarted","Data":"2baafcf49b80803f7ec4ead12953f273c26abf78b33796eb5a3378cfdddca1d1"}
May 25 13:07:22.843503 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:22.843477    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-vc5f4" event={"ID":"127a81c2-9752-4c7f-979c-d5aa0d2a86b5","Type":"ContainerStarted","Data":"2058dfdef986cb17cca2848a3f4cb61a801a028d62c111ba40c0a80f9703646a"}
May 25 13:07:30.323511 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.323469    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96" podUID="080f2207-b1de-4749-ab1d-a8e445abf8fa" containerName="registry" containerID="cri-o://2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1" gracePeriod=30
May 25 13:07:30.561540 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.561519    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:07:30.579018 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.578970    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcrdw\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-kube-api-access-bcrdw\") pod \"080f2207-b1de-4749-ab1d-a8e445abf8fa\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") "
May 25 13:07:30.579018 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.579012    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-bound-sa-token\") pod \"080f2207-b1de-4749-ab1d-a8e445abf8fa\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") "
May 25 13:07:30.579139 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.579040    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-certificates\") pod \"080f2207-b1de-4749-ab1d-a8e445abf8fa\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") "
May 25 13:07:30.579139 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.579081    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-image-registry-private-configuration\") pod \"080f2207-b1de-4749-ab1d-a8e445abf8fa\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") "
May 25 13:07:30.579139 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.579126    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-trusted-ca\") pod \"080f2207-b1de-4749-ab1d-a8e445abf8fa\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") "
May 25 13:07:30.579262 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.579162    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-installation-pull-secrets\") pod \"080f2207-b1de-4749-ab1d-a8e445abf8fa\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") "
May 25 13:07:30.579262 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.579194    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/080f2207-b1de-4749-ab1d-a8e445abf8fa-ca-trust-extracted\") pod \"080f2207-b1de-4749-ab1d-a8e445abf8fa\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") "
May 25 13:07:30.579262 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.579235    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") pod \"080f2207-b1de-4749-ab1d-a8e445abf8fa\" (UID: \"080f2207-b1de-4749-ab1d-a8e445abf8fa\") "
May 25 13:07:30.579672 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.579505    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "080f2207-b1de-4749-ab1d-a8e445abf8fa" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:30.579672 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.579658    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "080f2207-b1de-4749-ab1d-a8e445abf8fa" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:30.581326 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.581303    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "080f2207-b1de-4749-ab1d-a8e445abf8fa" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:30.581454 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.581429    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-kube-api-access-bcrdw" (OuterVolumeSpecName: "kube-api-access-bcrdw") pod "080f2207-b1de-4749-ab1d-a8e445abf8fa" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa"). InnerVolumeSpecName "kube-api-access-bcrdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:30.581631 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.581605    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "080f2207-b1de-4749-ab1d-a8e445abf8fa" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:07:30.581724 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.581676    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "080f2207-b1de-4749-ab1d-a8e445abf8fa" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:07:30.581783 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.581752    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "080f2207-b1de-4749-ab1d-a8e445abf8fa" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:30.589423 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.589393    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080f2207-b1de-4749-ab1d-a8e445abf8fa-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "080f2207-b1de-4749-ab1d-a8e445abf8fa" (UID: "080f2207-b1de-4749-ab1d-a8e445abf8fa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:07:30.679892 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.679866    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-bound-sa-token\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:07:30.679997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.679896    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-certificates\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:07:30.679997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.679916    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-image-registry-private-configuration\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:07:30.679997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.679932    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/080f2207-b1de-4749-ab1d-a8e445abf8fa-trusted-ca\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:07:30.679997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.679946    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/080f2207-b1de-4749-ab1d-a8e445abf8fa-installation-pull-secrets\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:07:30.679997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.679958    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/080f2207-b1de-4749-ab1d-a8e445abf8fa-ca-trust-extracted\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:07:30.679997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.679970    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-registry-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:07:30.679997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.679983    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrdw\" (UniqueName: \"kubernetes.io/projected/080f2207-b1de-4749-ab1d-a8e445abf8fa-kube-api-access-bcrdw\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:07:30.869271 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.869204    2573 generic.go:358] "Generic (PLEG): container finished" podID="080f2207-b1de-4749-ab1d-a8e445abf8fa" containerID="2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1" exitCode=0
May 25 13:07:30.869353 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.869271    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96"
May 25 13:07:30.869401 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.869268    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96" event={"ID":"080f2207-b1de-4749-ab1d-a8e445abf8fa","Type":"ContainerDied","Data":"2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1"}
May 25 13:07:30.869401 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.869384    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb44d44c7-qzx96" event={"ID":"080f2207-b1de-4749-ab1d-a8e445abf8fa","Type":"ContainerDied","Data":"9b81af125950005fef0d36cd25dda313be7ef851f6c8de77a19567ceed3711e1"}
May 25 13:07:30.869461 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.869406    2573 scope.go:117] "RemoveContainer" containerID="2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1"
May 25 13:07:30.878456 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.878434    2573 scope.go:117] "RemoveContainer" containerID="2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1"
May 25 13:07:30.878770 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:07:30.878749    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1\": container with ID starting with 2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1 not found: ID does not exist" containerID="2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1"
May 25 13:07:30.878832 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.878778    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1"} err="failed to get container status \"2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1\": rpc error: code = NotFound desc = could not find container \"2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1\": container with ID starting with 2d0626aa6fe35a9753ef70baa623c65639cbfc98a0c3927a6902e7432267faa1 not found: ID does not exist"
May 25 13:07:30.894902 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.893685    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb44d44c7-qzx96"]
May 25 13:07:30.897839 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:30.897813    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fb44d44c7-qzx96"]
May 25 13:07:31.337004 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:31.336974    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="080f2207-b1de-4749-ab1d-a8e445abf8fa" path="/var/lib/kubelet/pods/080f2207-b1de-4749-ab1d-a8e445abf8fa/volumes"
May 25 13:07:31.873723 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:31.873689    2573 generic.go:358] "Generic (PLEG): container finished" podID="a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8" containerID="83fcdcfd0a3e7b817c87966170c766ac463f2c3f37f87aa9e822778ca794a738" exitCode=0
May 25 13:07:31.873911 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:31.873780    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql" event={"ID":"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8","Type":"ContainerDied","Data":"83fcdcfd0a3e7b817c87966170c766ac463f2c3f37f87aa9e822778ca794a738"}
May 25 13:07:31.874248 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:31.874215    2573 scope.go:117] "RemoveContainer" containerID="83fcdcfd0a3e7b817c87966170c766ac463f2c3f37f87aa9e822778ca794a738"
May 25 13:07:32.880305 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:07:32.880266    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-8kgql" event={"ID":"a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8","Type":"ContainerStarted","Data":"d07502ad01dc9c4bb5f32daed3ce9e19dc0045714bb748cadf35ea7c708050ce"}
May 25 13:10:33.244540 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:10:33.244496    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:10:33.246054 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:10:33.246023    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:10:33.258739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:10:33.256108    2573 kubelet.go:1628] "Image garbage collection succeeded"
May 25 13:15:33.268625 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:15:33.268597    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:15:33.271657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:15:33.271632    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:20:05.533533 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.533495    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"]
May 25 13:20:05.534058 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.533777    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080f2207-b1de-4749-ab1d-a8e445abf8fa" containerName="registry"
May 25 13:20:05.534058 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.533789    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="080f2207-b1de-4749-ab1d-a8e445abf8fa" containerName="registry"
May 25 13:20:05.534058 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.533860    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="080f2207-b1de-4749-ab1d-a8e445abf8fa" containerName="registry"
May 25 13:20:05.536770 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.536750    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"
May 25 13:20:05.539049 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.539023    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
May 25 13:20:05.539176 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.539035    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-4csnv\""
May 25 13:20:05.539448 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.539428    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
May 25 13:20:05.546760 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.546736    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"]
May 25 13:20:05.616887 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.616859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdgt\" (UniqueName: \"kubernetes.io/projected/148eaafa-4e92-4803-a030-1ee1b7089db4-kube-api-access-bwdgt\") pod \"cert-manager-operator-controller-manager-54b9655956-2vsd8\" (UID: \"148eaafa-4e92-4803-a030-1ee1b7089db4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"
May 25 13:20:05.616990 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.616912    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/148eaafa-4e92-4803-a030-1ee1b7089db4-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-2vsd8\" (UID: \"148eaafa-4e92-4803-a030-1ee1b7089db4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"
May 25 13:20:05.717351 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.717327    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdgt\" (UniqueName: \"kubernetes.io/projected/148eaafa-4e92-4803-a030-1ee1b7089db4-kube-api-access-bwdgt\") pod \"cert-manager-operator-controller-manager-54b9655956-2vsd8\" (UID: \"148eaafa-4e92-4803-a030-1ee1b7089db4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"
May 25 13:20:05.717439 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.717367    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/148eaafa-4e92-4803-a030-1ee1b7089db4-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-2vsd8\" (UID: \"148eaafa-4e92-4803-a030-1ee1b7089db4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"
May 25 13:20:05.717682 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.717667    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/148eaafa-4e92-4803-a030-1ee1b7089db4-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-2vsd8\" (UID: \"148eaafa-4e92-4803-a030-1ee1b7089db4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"
May 25 13:20:05.726231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.726206    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdgt\" (UniqueName: \"kubernetes.io/projected/148eaafa-4e92-4803-a030-1ee1b7089db4-kube-api-access-bwdgt\") pod \"cert-manager-operator-controller-manager-54b9655956-2vsd8\" (UID: \"148eaafa-4e92-4803-a030-1ee1b7089db4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"
May 25 13:20:05.847041 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.846984    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"
May 25 13:20:05.975007 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.974979    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8"]
May 25 13:20:05.976979 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:20:05.976949    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148eaafa_4e92_4803_a030_1ee1b7089db4.slice/crio-262fbe3a1a6e04238e55937b4fca343167a392f01a4a47ee3d9d3b9454b49e1e WatchSource:0}: Error finding container 262fbe3a1a6e04238e55937b4fca343167a392f01a4a47ee3d9d3b9454b49e1e: Status 404 returned error can't find the container with id 262fbe3a1a6e04238e55937b4fca343167a392f01a4a47ee3d9d3b9454b49e1e
May 25 13:20:05.979412 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:05.979392    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:20:06.984031 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:06.983985    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8" event={"ID":"148eaafa-4e92-4803-a030-1ee1b7089db4","Type":"ContainerStarted","Data":"262fbe3a1a6e04238e55937b4fca343167a392f01a4a47ee3d9d3b9454b49e1e"}
May 25 13:20:07.990688 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:07.990651    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8" event={"ID":"148eaafa-4e92-4803-a030-1ee1b7089db4","Type":"ContainerStarted","Data":"a4d4a2ad1a62e3d61cbdb6260bcd215e04b60e5ff4d20667d31a629383065bd6"}
May 25 13:20:08.021691 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:08.021340    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-2vsd8" podStartSLOduration=1.4889904330000001 podStartE2EDuration="3.021321222s" podCreationTimestamp="2026-05-25 13:20:05 +0000 UTC" firstStartedPulling="2026-05-25 13:20:05.979565804 +0000 UTC m=+873.241168612" lastFinishedPulling="2026-05-25 13:20:07.511896586 +0000 UTC m=+874.773499401" observedRunningTime="2026-05-25 13:20:08.020004671 +0000 UTC m=+875.281607503" watchObservedRunningTime="2026-05-25 13:20:08.021321222 +0000 UTC m=+875.282924054"
May 25 13:20:20.793233 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:20.793151    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-79c8d999ff-fwxxt"]
May 25 13:20:20.796321 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:20.796296    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-fwxxt"
May 25 13:20:20.798944 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:20.798919    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 25 13:20:20.799140 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:20.799118    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-m74nk\""
May 25 13:20:20.799712 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:20.799696    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 25 13:20:20.840101 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:20.840077    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-fwxxt"]
May 25 13:20:20.923029 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:20.923008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfrg\" (UniqueName: \"kubernetes.io/projected/c708bc2c-1675-46e4-b215-a9f19f962a29-kube-api-access-6sfrg\") pod \"cert-manager-79c8d999ff-fwxxt\" (UID: \"c708bc2c-1675-46e4-b215-a9f19f962a29\") " pod="cert-manager/cert-manager-79c8d999ff-fwxxt"
May 25 13:20:20.923135 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:20.923051    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c708bc2c-1675-46e4-b215-a9f19f962a29-bound-sa-token\") pod \"cert-manager-79c8d999ff-fwxxt\" (UID: \"c708bc2c-1675-46e4-b215-a9f19f962a29\") " pod="cert-manager/cert-manager-79c8d999ff-fwxxt"
May 25 13:20:21.024305 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:21.024282    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfrg\" (UniqueName: \"kubernetes.io/projected/c708bc2c-1675-46e4-b215-a9f19f962a29-kube-api-access-6sfrg\") pod \"cert-manager-79c8d999ff-fwxxt\" (UID: \"c708bc2c-1675-46e4-b215-a9f19f962a29\") " pod="cert-manager/cert-manager-79c8d999ff-fwxxt"
May 25 13:20:21.024416 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:21.024320    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c708bc2c-1675-46e4-b215-a9f19f962a29-bound-sa-token\") pod \"cert-manager-79c8d999ff-fwxxt\" (UID: \"c708bc2c-1675-46e4-b215-a9f19f962a29\") " pod="cert-manager/cert-manager-79c8d999ff-fwxxt"
May 25 13:20:21.035990 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:21.035963    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfrg\" (UniqueName: \"kubernetes.io/projected/c708bc2c-1675-46e4-b215-a9f19f962a29-kube-api-access-6sfrg\") pod \"cert-manager-79c8d999ff-fwxxt\" (UID: \"c708bc2c-1675-46e4-b215-a9f19f962a29\") " pod="cert-manager/cert-manager-79c8d999ff-fwxxt"
May 25 13:20:21.036279 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:21.036257    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c708bc2c-1675-46e4-b215-a9f19f962a29-bound-sa-token\") pod \"cert-manager-79c8d999ff-fwxxt\" (UID: \"c708bc2c-1675-46e4-b215-a9f19f962a29\") " pod="cert-manager/cert-manager-79c8d999ff-fwxxt"
May 25 13:20:21.123617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:21.123538    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-fwxxt"
May 25 13:20:21.243679 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:21.243646    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-fwxxt"]
May 25 13:20:21.246309 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:20:21.246281    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc708bc2c_1675_46e4_b215_a9f19f962a29.slice/crio-ac131293e2fd0c49a542656fae157a4f9e09391a7892afe151d093a938ec8d14 WatchSource:0}: Error finding container ac131293e2fd0c49a542656fae157a4f9e09391a7892afe151d093a938ec8d14: Status 404 returned error can't find the container with id ac131293e2fd0c49a542656fae157a4f9e09391a7892afe151d093a938ec8d14
May 25 13:20:22.040824 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:22.040773    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-fwxxt" event={"ID":"c708bc2c-1675-46e4-b215-a9f19f962a29","Type":"ContainerStarted","Data":"ac131293e2fd0c49a542656fae157a4f9e09391a7892afe151d093a938ec8d14"}
May 25 13:20:24.048238 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.048206    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-fwxxt" event={"ID":"c708bc2c-1675-46e4-b215-a9f19f962a29","Type":"ContainerStarted","Data":"6d8e45ffb7fb3626837135c22cff84a371c8b47585b4bc1fe80543eab9e3234b"}
May 25 13:20:24.066875 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.066831    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-79c8d999ff-fwxxt" podStartSLOduration=1.9077477630000002 podStartE2EDuration="4.06681744s" podCreationTimestamp="2026-05-25 13:20:20 +0000 UTC" firstStartedPulling="2026-05-25 13:20:21.24802538 +0000 UTC m=+888.509628192" lastFinishedPulling="2026-05-25 13:20:23.40709504 +0000 UTC m=+890.668697869" observedRunningTime="2026-05-25 13:20:24.065840148 +0000 UTC m=+891.327442979" watchObservedRunningTime="2026-05-25 13:20:24.06681744 +0000 UTC m=+891.328420270"
May 25 13:20:24.681020 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.680990    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"]
May 25 13:20:24.684099 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.684084    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"
May 25 13:20:24.694603 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.694580    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-tr82g\""
May 25 13:20:24.694754 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.694585    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 25 13:20:24.694754 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.694585    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 25 13:20:24.713461 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.713435    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"]
May 25 13:20:24.750720 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.750696    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6kc\" (UniqueName: \"kubernetes.io/projected/226140ab-8d9e-4b33-a30e-45779b3dbb1a-kube-api-access-bx6kc\") pod \"openshift-lws-operator-bfc7f696d-8xw5x\" (UID: \"226140ab-8d9e-4b33-a30e-45779b3dbb1a\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"
May 25 13:20:24.750813 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.750728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/226140ab-8d9e-4b33-a30e-45779b3dbb1a-tmp\") pod \"openshift-lws-operator-bfc7f696d-8xw5x\" (UID: \"226140ab-8d9e-4b33-a30e-45779b3dbb1a\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"
May 25 13:20:24.852001 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.851979    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6kc\" (UniqueName: \"kubernetes.io/projected/226140ab-8d9e-4b33-a30e-45779b3dbb1a-kube-api-access-bx6kc\") pod \"openshift-lws-operator-bfc7f696d-8xw5x\" (UID: \"226140ab-8d9e-4b33-a30e-45779b3dbb1a\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"
May 25 13:20:24.852074 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.852007    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/226140ab-8d9e-4b33-a30e-45779b3dbb1a-tmp\") pod \"openshift-lws-operator-bfc7f696d-8xw5x\" (UID: \"226140ab-8d9e-4b33-a30e-45779b3dbb1a\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"
May 25 13:20:24.852303 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.852290    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/226140ab-8d9e-4b33-a30e-45779b3dbb1a-tmp\") pod \"openshift-lws-operator-bfc7f696d-8xw5x\" (UID: \"226140ab-8d9e-4b33-a30e-45779b3dbb1a\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"
May 25 13:20:24.860265 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.860245    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6kc\" (UniqueName: \"kubernetes.io/projected/226140ab-8d9e-4b33-a30e-45779b3dbb1a-kube-api-access-bx6kc\") pod \"openshift-lws-operator-bfc7f696d-8xw5x\" (UID: \"226140ab-8d9e-4b33-a30e-45779b3dbb1a\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"
May 25 13:20:24.993310 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:24.993291    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"
May 25 13:20:25.119603 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:25.119573    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x"]
May 25 13:20:25.122308 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:20:25.122276    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod226140ab_8d9e_4b33_a30e_45779b3dbb1a.slice/crio-9794053e323bb6c72f40eb9da58b5654185561aaa1f41e8af6334c1954b75a05 WatchSource:0}: Error finding container 9794053e323bb6c72f40eb9da58b5654185561aaa1f41e8af6334c1954b75a05: Status 404 returned error can't find the container with id 9794053e323bb6c72f40eb9da58b5654185561aaa1f41e8af6334c1954b75a05
May 25 13:20:26.057180 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:26.057137    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x" event={"ID":"226140ab-8d9e-4b33-a30e-45779b3dbb1a","Type":"ContainerStarted","Data":"9794053e323bb6c72f40eb9da58b5654185561aaa1f41e8af6334c1954b75a05"}
May 25 13:20:28.066117 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:28.066078    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x" event={"ID":"226140ab-8d9e-4b33-a30e-45779b3dbb1a","Type":"ContainerStarted","Data":"d528ffdd37a90056d64bfb9d5129cf581ceee56733a24e6ca974c7ea343a048c"}
May 25 13:20:28.082781 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:28.082726    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-8xw5x" podStartSLOduration=1.641032047 podStartE2EDuration="4.082708381s" podCreationTimestamp="2026-05-25 13:20:24 +0000 UTC" firstStartedPulling="2026-05-25 13:20:25.123753143 +0000 UTC m=+892.385355956" lastFinishedPulling="2026-05-25 13:20:27.565429482 +0000 UTC m=+894.827032290" observedRunningTime="2026-05-25 13:20:28.081621496 +0000 UTC m=+895.343224331" watchObservedRunningTime="2026-05-25 13:20:28.082708381 +0000 UTC m=+895.344311215"
May 25 13:20:33.291707 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:33.291677    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:20:33.292095 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:33.292073    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:20:56.879262 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:56.879225    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"]
May 25 13:20:56.882288 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:56.882272    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:56.886396 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:56.886362    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 25 13:20:56.886805 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:56.886783    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 25 13:20:56.887362 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:56.887348    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 25 13:20:56.887415 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:56.887352    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dbmdq\""
May 25 13:20:56.899830 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:56.899804    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"]
May 25 13:20:57.066463 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.066432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-metrics-cert\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.066621 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.066480    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l52b\" (UniqueName: \"kubernetes.io/projected/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-kube-api-access-7l52b\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.066621 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.066596    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-manager-config\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.066621 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.066619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-cert\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.167568 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.167492    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-manager-config\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.167568 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.167518    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-cert\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.167568 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.167566    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-metrics-cert\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.167737 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.167594    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7l52b\" (UniqueName: \"kubernetes.io/projected/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-kube-api-access-7l52b\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.168184 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.168163    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-manager-config\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.170011 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.169991    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-cert\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.170084 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.170029    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-metrics-cert\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.182623 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.182602    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l52b\" (UniqueName: \"kubernetes.io/projected/b83b4cb1-3e33-459b-b6f9-04e4e7315c6d-kube-api-access-7l52b\") pod \"lws-controller-manager-6f6cfd7d7-p4xl7\" (UID: \"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.191361 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.191333    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:57.350345 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:57.350253    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"]
May 25 13:20:57.352333 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:20:57.352308    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83b4cb1_3e33_459b_b6f9_04e4e7315c6d.slice/crio-807ec947b578a94dc69fc69afc20b72721f2289165fbc0ffb95f3a8da6d6c59a WatchSource:0}: Error finding container 807ec947b578a94dc69fc69afc20b72721f2289165fbc0ffb95f3a8da6d6c59a: Status 404 returned error can't find the container with id 807ec947b578a94dc69fc69afc20b72721f2289165fbc0ffb95f3a8da6d6c59a
May 25 13:20:58.162586 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:58.162537    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7" event={"ID":"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d","Type":"ContainerStarted","Data":"807ec947b578a94dc69fc69afc20b72721f2289165fbc0ffb95f3a8da6d6c59a"}
May 25 13:20:59.166767 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:59.166735    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7" event={"ID":"b83b4cb1-3e33-459b-b6f9-04e4e7315c6d","Type":"ContainerStarted","Data":"b6d77e5132a00b8c2eb9c0e9dbeefe34c122224b12a901a5756534a3c7aa68de"}
May 25 13:20:59.167112 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:59.166786    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:20:59.190736 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:20:59.190669    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7" podStartSLOduration=1.594465869 podStartE2EDuration="3.190623638s" podCreationTimestamp="2026-05-25 13:20:56 +0000 UTC" firstStartedPulling="2026-05-25 13:20:57.354116652 +0000 UTC m=+924.615719461" lastFinishedPulling="2026-05-25 13:20:58.950274418 +0000 UTC m=+926.211877230" observedRunningTime="2026-05-25 13:20:59.188459008 +0000 UTC m=+926.450061851" watchObservedRunningTime="2026-05-25 13:20:59.190623638 +0000 UTC m=+926.452226469"
May 25 13:21:10.172143 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:10.172110    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-p4xl7"
May 25 13:21:50.766208 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.766176    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"]
May 25 13:21:50.769431 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.769416    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:50.785670 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.785654    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
May 25 13:21:50.786653 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.786618    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 25 13:21:50.786776 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.786762    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
May 25 13:21:50.787520 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.787504    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 25 13:21:50.787748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.787734    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p5cd7\""
May 25 13:21:50.804395 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.804374    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"]
May 25 13:21:50.836353 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.836327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggwwc\" (UniqueName: \"kubernetes.io/projected/1cacaf2f-89c3-4de9-96a6-656d11ffa06a-kube-api-access-ggwwc\") pod \"kuadrant-console-plugin-6c886788f8-26l97\" (UID: \"1cacaf2f-89c3-4de9-96a6-656d11ffa06a\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:50.836442 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.836400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1cacaf2f-89c3-4de9-96a6-656d11ffa06a-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-26l97\" (UID: \"1cacaf2f-89c3-4de9-96a6-656d11ffa06a\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:50.836442 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.836419    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cacaf2f-89c3-4de9-96a6-656d11ffa06a-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-26l97\" (UID: \"1cacaf2f-89c3-4de9-96a6-656d11ffa06a\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:50.937140 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.937121    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1cacaf2f-89c3-4de9-96a6-656d11ffa06a-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-26l97\" (UID: \"1cacaf2f-89c3-4de9-96a6-656d11ffa06a\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:50.937241 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.937148    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cacaf2f-89c3-4de9-96a6-656d11ffa06a-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-26l97\" (UID: \"1cacaf2f-89c3-4de9-96a6-656d11ffa06a\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:50.937241 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.937172    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggwwc\" (UniqueName: \"kubernetes.io/projected/1cacaf2f-89c3-4de9-96a6-656d11ffa06a-kube-api-access-ggwwc\") pod \"kuadrant-console-plugin-6c886788f8-26l97\" (UID: \"1cacaf2f-89c3-4de9-96a6-656d11ffa06a\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:50.937901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.937880    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1cacaf2f-89c3-4de9-96a6-656d11ffa06a-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-26l97\" (UID: \"1cacaf2f-89c3-4de9-96a6-656d11ffa06a\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:50.939521 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.939503    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cacaf2f-89c3-4de9-96a6-656d11ffa06a-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-26l97\" (UID: \"1cacaf2f-89c3-4de9-96a6-656d11ffa06a\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:50.956685 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:50.956658    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggwwc\" (UniqueName: \"kubernetes.io/projected/1cacaf2f-89c3-4de9-96a6-656d11ffa06a-kube-api-access-ggwwc\") pod \"kuadrant-console-plugin-6c886788f8-26l97\" (UID: \"1cacaf2f-89c3-4de9-96a6-656d11ffa06a\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:51.078738 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:51.078687    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"
May 25 13:21:51.228797 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:21:51.228767    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cacaf2f_89c3_4de9_96a6_656d11ffa06a.slice/crio-b63637c8a96883322b70a6f796b7b49a1abf0df378858e4f32587f789e7e2517 WatchSource:0}: Error finding container b63637c8a96883322b70a6f796b7b49a1abf0df378858e4f32587f789e7e2517: Status 404 returned error can't find the container with id b63637c8a96883322b70a6f796b7b49a1abf0df378858e4f32587f789e7e2517
May 25 13:21:51.240218 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:51.240193    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97"]
May 25 13:21:51.338234 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:51.338180    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97" event={"ID":"1cacaf2f-89c3-4de9-96a6-656d11ffa06a","Type":"ContainerStarted","Data":"b63637c8a96883322b70a6f796b7b49a1abf0df378858e4f32587f789e7e2517"}
May 25 13:21:57.359531 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:57.359497    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97" event={"ID":"1cacaf2f-89c3-4de9-96a6-656d11ffa06a","Type":"ContainerStarted","Data":"23702d36c6a80cb7fca425a27246aee2f8ee59eb6adbbc6ba395c24877a48db9"}
May 25 13:21:57.392617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:21:57.392576    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-26l97" podStartSLOduration=1.777389727 podStartE2EDuration="7.392539404s" podCreationTimestamp="2026-05-25 13:21:50 +0000 UTC" firstStartedPulling="2026-05-25 13:21:51.229987866 +0000 UTC m=+978.491590678" lastFinishedPulling="2026-05-25 13:21:56.845137546 +0000 UTC m=+984.106740355" observedRunningTime="2026-05-25 13:21:57.391323837 +0000 UTC m=+984.652926669" watchObservedRunningTime="2026-05-25 13:21:57.392539404 +0000 UTC m=+984.654142236"
May 25 13:22:34.950154 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:34.950120    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-kkz9n"]
May 25 13:22:34.953252 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:34.953236    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n"
May 25 13:22:34.955263 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:34.955245    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 25 13:22:34.962445 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:34.962426    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-kkz9n"]
May 25 13:22:34.990939 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:34.990919    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-kkz9n"]
May 25 13:22:35.060061 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:35.060041    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hkcw\" (UniqueName: \"kubernetes.io/projected/65f6e7db-0b43-4ce5-a23f-030ad00c428e-kube-api-access-5hkcw\") pod \"limitador-limitador-67566c68b4-kkz9n\" (UID: \"65f6e7db-0b43-4ce5-a23f-030ad00c428e\") " pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n"
May 25 13:22:35.060176 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:35.060072    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65f6e7db-0b43-4ce5-a23f-030ad00c428e-config-file\") pod \"limitador-limitador-67566c68b4-kkz9n\" (UID: \"65f6e7db-0b43-4ce5-a23f-030ad00c428e\") " pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n"
May 25 13:22:35.161364 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:35.161338    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5hkcw\" (UniqueName: \"kubernetes.io/projected/65f6e7db-0b43-4ce5-a23f-030ad00c428e-kube-api-access-5hkcw\") pod \"limitador-limitador-67566c68b4-kkz9n\" (UID: \"65f6e7db-0b43-4ce5-a23f-030ad00c428e\") " pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n"
May 25 13:22:35.161454 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:35.161374    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65f6e7db-0b43-4ce5-a23f-030ad00c428e-config-file\") pod \"limitador-limitador-67566c68b4-kkz9n\" (UID: \"65f6e7db-0b43-4ce5-a23f-030ad00c428e\") " pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n"
May 25 13:22:35.161941 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:35.161923    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65f6e7db-0b43-4ce5-a23f-030ad00c428e-config-file\") pod \"limitador-limitador-67566c68b4-kkz9n\" (UID: \"65f6e7db-0b43-4ce5-a23f-030ad00c428e\") " pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n"
May 25 13:22:35.168686 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:35.168667    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hkcw\" (UniqueName: \"kubernetes.io/projected/65f6e7db-0b43-4ce5-a23f-030ad00c428e-kube-api-access-5hkcw\") pod \"limitador-limitador-67566c68b4-kkz9n\" (UID: \"65f6e7db-0b43-4ce5-a23f-030ad00c428e\") " pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n"
May 25 13:22:35.263897 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:35.263874    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n"
May 25 13:22:35.392058 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:35.392033    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-kkz9n"]
May 25 13:22:35.394834 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:22:35.394810    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f6e7db_0b43_4ce5_a23f_030ad00c428e.slice/crio-93ae961323625c6b0ac50f1c879c64327a5860d945aa3a22b61ac25f7703acaf WatchSource:0}: Error finding container 93ae961323625c6b0ac50f1c879c64327a5860d945aa3a22b61ac25f7703acaf: Status 404 returned error can't find the container with id 93ae961323625c6b0ac50f1c879c64327a5860d945aa3a22b61ac25f7703acaf
May 25 13:22:35.471748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:35.471719    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n" event={"ID":"65f6e7db-0b43-4ce5-a23f-030ad00c428e","Type":"ContainerStarted","Data":"93ae961323625c6b0ac50f1c879c64327a5860d945aa3a22b61ac25f7703acaf"}
May 25 13:22:36.475787 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:36.475754    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n" event={"ID":"65f6e7db-0b43-4ce5-a23f-030ad00c428e","Type":"ContainerStarted","Data":"0f57127cdd3611afe3d44b5b059b04f220dc11e6dc55d8d713c2ba7e7d2fabfe"}
May 25 13:22:36.476062 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:36.475869    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n"
May 25 13:22:36.492306 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:36.492264    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n" podStartSLOduration=1.482196721 podStartE2EDuration="2.492252005s" podCreationTimestamp="2026-05-25 13:22:34 +0000 UTC" firstStartedPulling="2026-05-25 13:22:35.39709407 +0000 UTC m=+1022.658696878" lastFinishedPulling="2026-05-25 13:22:36.40714935 +0000 UTC m=+1023.668752162" observedRunningTime="2026-05-25 13:22:36.490572939 +0000 UTC m=+1023.752175785" watchObservedRunningTime="2026-05-25 13:22:36.492252005 +0000 UTC m=+1023.753854835"
May 25 13:22:47.480573 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:22:47.480515    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-67566c68b4-kkz9n"
May 25 13:25:25.812329 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.812293    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"]
May 25 13:25:25.816621 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.816599    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:25.819194 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.819165    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 13:25:25.819335 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.819243    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:25:25.819335 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.819247    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:25:25.819765 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.819751    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
May 25 13:25:25.828035 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.828008    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"]
May 25 13:25:25.966313 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.966279    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:25.966430 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.966325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27f78\" (UniqueName: \"kubernetes.io/projected/d052fdfd-3229-47e1-95d0-a3db5599f192-kube-api-access-27f78\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:25.966430 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.966347    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:25.966430 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.966371    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d052fdfd-3229-47e1-95d0-a3db5599f192-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:25.966430 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.966400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:25.966430 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.966430    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:25.966656 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:25.966452    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067131 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067075    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067131 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067114    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067298 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067136    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067298 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067166    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067298 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067193    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-27f78\" (UniqueName: \"kubernetes.io/projected/d052fdfd-3229-47e1-95d0-a3db5599f192-kube-api-access-27f78\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067298 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067213    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067298 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067233    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d052fdfd-3229-47e1-95d0-a3db5599f192-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067582 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067537    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067648 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067588    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067703 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067650    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.067742 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.067712    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.069567 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.069533    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.069847 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.069830    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d052fdfd-3229-47e1-95d0-a3db5599f192-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.075450 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.075427    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27f78\" (UniqueName: \"kubernetes.io/projected/d052fdfd-3229-47e1-95d0-a3db5599f192-kube-api-access-27f78\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.129418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.129396    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:26.259034 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.259011    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"]
May 25 13:25:26.261168 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:25:26.261145    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd052fdfd_3229_47e1_95d0_a3db5599f192.slice/crio-6565a549d0fa0f30f44b056f433c5df5ed77e3d6dd2e31d2ffa5ae29de6e34c9 WatchSource:0}: Error finding container 6565a549d0fa0f30f44b056f433c5df5ed77e3d6dd2e31d2ffa5ae29de6e34c9: Status 404 returned error can't find the container with id 6565a549d0fa0f30f44b056f433c5df5ed77e3d6dd2e31d2ffa5ae29de6e34c9
May 25 13:25:26.263200 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:26.263186    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:25:27.024476 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:27.024429    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j" event={"ID":"d052fdfd-3229-47e1-95d0-a3db5599f192","Type":"ContainerStarted","Data":"6565a549d0fa0f30f44b056f433c5df5ed77e3d6dd2e31d2ffa5ae29de6e34c9"}
May 25 13:25:30.036395 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:30.036360    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j" event={"ID":"d052fdfd-3229-47e1-95d0-a3db5599f192","Type":"ContainerStarted","Data":"618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f"}
May 25 13:25:33.314992 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:33.314964    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:25:33.315438 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:33.315169    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:25:34.052046 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:34.052006    2573 generic.go:358] "Generic (PLEG): container finished" podID="d052fdfd-3229-47e1-95d0-a3db5599f192" containerID="618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f" exitCode=0
May 25 13:25:34.052254 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:34.052083    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j" event={"ID":"d052fdfd-3229-47e1-95d0-a3db5599f192","Type":"ContainerDied","Data":"618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f"}
May 25 13:25:36.059957 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:36.059920    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j" event={"ID":"d052fdfd-3229-47e1-95d0-a3db5599f192","Type":"ContainerStarted","Data":"8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1"}
May 25 13:25:36.077990 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:36.077940    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j" podStartSLOduration=2.121641109 podStartE2EDuration="11.07792552s" podCreationTimestamp="2026-05-25 13:25:25 +0000 UTC" firstStartedPulling="2026-05-25 13:25:26.263308562 +0000 UTC m=+1193.524911371" lastFinishedPulling="2026-05-25 13:25:35.21959297 +0000 UTC m=+1202.481195782" observedRunningTime="2026-05-25 13:25:36.07607832 +0000 UTC m=+1203.337681150" watchObservedRunningTime="2026-05-25 13:25:36.07792552 +0000 UTC m=+1203.339528351"
May 25 13:25:36.129908 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:36.129885    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:36.130003 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:36.129927    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:36.142200 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:36.142183    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:37.074565 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:37.074517    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:38.925018 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:38.924985    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"]
May 25 13:25:39.181707 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:25:39.181634    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
May 25 13:25:39.181846 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:25:39.181712    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d052fdfd-3229-47e1-95d0-a3db5599f192-tls-certs podName:d052fdfd-3229-47e1-95d0-a3db5599f192 nodeName:}" failed. No retries permitted until 2026-05-25 13:25:39.681695833 +0000 UTC m=+1206.943298643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d052fdfd-3229-47e1-95d0-a3db5599f192-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j" (UID: "d052fdfd-3229-47e1-95d0-a3db5599f192") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
May 25 13:25:39.685286 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:25:39.685256    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
May 25 13:25:39.685460 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:25:39.685325    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d052fdfd-3229-47e1-95d0-a3db5599f192-tls-certs podName:d052fdfd-3229-47e1-95d0-a3db5599f192 nodeName:}" failed. No retries permitted until 2026-05-25 13:25:40.685310967 +0000 UTC m=+1207.946913777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d052fdfd-3229-47e1-95d0-a3db5599f192-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j" (UID: "d052fdfd-3229-47e1-95d0-a3db5599f192") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
May 25 13:25:40.076944 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.076906    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j" podUID="d052fdfd-3229-47e1-95d0-a3db5599f192" containerName="main" containerID="cri-o://8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1" gracePeriod=30
May 25 13:25:40.349739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.349716    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:40.491423 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491394    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-kserve-provision-location\") pod \"d052fdfd-3229-47e1-95d0-a3db5599f192\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") "
May 25 13:25:40.491533 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491447    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-dshm\") pod \"d052fdfd-3229-47e1-95d0-a3db5599f192\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") "
May 25 13:25:40.491533 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491464    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-tmp-dir\") pod \"d052fdfd-3229-47e1-95d0-a3db5599f192\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") "
May 25 13:25:40.491671 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491493    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-home\") pod \"d052fdfd-3229-47e1-95d0-a3db5599f192\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") "
May 25 13:25:40.491671 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491629    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27f78\" (UniqueName: \"kubernetes.io/projected/d052fdfd-3229-47e1-95d0-a3db5599f192-kube-api-access-27f78\") pod \"d052fdfd-3229-47e1-95d0-a3db5599f192\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") "
May 25 13:25:40.491671 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491655    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d052fdfd-3229-47e1-95d0-a3db5599f192-tls-certs\") pod \"d052fdfd-3229-47e1-95d0-a3db5599f192\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") "
May 25 13:25:40.491843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491683    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-model-cache\") pod \"d052fdfd-3229-47e1-95d0-a3db5599f192\" (UID: \"d052fdfd-3229-47e1-95d0-a3db5599f192\") "
May 25 13:25:40.491843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491699    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-home" (OuterVolumeSpecName: "home") pod "d052fdfd-3229-47e1-95d0-a3db5599f192" (UID: "d052fdfd-3229-47e1-95d0-a3db5599f192"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:25:40.491843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491787    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d052fdfd-3229-47e1-95d0-a3db5599f192" (UID: "d052fdfd-3229-47e1-95d0-a3db5599f192"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:25:40.491950 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491910    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:25:40.491992 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491951    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:25:40.492035 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.491989    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-model-cache" (OuterVolumeSpecName: "model-cache") pod "d052fdfd-3229-47e1-95d0-a3db5599f192" (UID: "d052fdfd-3229-47e1-95d0-a3db5599f192"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:25:40.493923 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.493897    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d052fdfd-3229-47e1-95d0-a3db5599f192-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d052fdfd-3229-47e1-95d0-a3db5599f192" (UID: "d052fdfd-3229-47e1-95d0-a3db5599f192"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:25:40.494039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.493897    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-dshm" (OuterVolumeSpecName: "dshm") pod "d052fdfd-3229-47e1-95d0-a3db5599f192" (UID: "d052fdfd-3229-47e1-95d0-a3db5599f192"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:25:40.494039 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.493970    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d052fdfd-3229-47e1-95d0-a3db5599f192-kube-api-access-27f78" (OuterVolumeSpecName: "kube-api-access-27f78") pod "d052fdfd-3229-47e1-95d0-a3db5599f192" (UID: "d052fdfd-3229-47e1-95d0-a3db5599f192"). InnerVolumeSpecName "kube-api-access-27f78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:25:40.559033 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.559003    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d052fdfd-3229-47e1-95d0-a3db5599f192" (UID: "d052fdfd-3229-47e1-95d0-a3db5599f192"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:25:40.593175 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.593130    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27f78\" (UniqueName: \"kubernetes.io/projected/d052fdfd-3229-47e1-95d0-a3db5599f192-kube-api-access-27f78\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:25:40.593175 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.593149    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d052fdfd-3229-47e1-95d0-a3db5599f192-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:25:40.593175 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.593160    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:25:40.593175 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.593169    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:25:40.593338 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:40.593178    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d052fdfd-3229-47e1-95d0-a3db5599f192-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:25:41.081826 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.081798    2573 generic.go:358] "Generic (PLEG): container finished" podID="d052fdfd-3229-47e1-95d0-a3db5599f192" containerID="8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1" exitCode=0
May 25 13:25:41.082222 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.081866    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j" event={"ID":"d052fdfd-3229-47e1-95d0-a3db5599f192","Type":"ContainerDied","Data":"8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1"}
May 25 13:25:41.082222 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.081905    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j" event={"ID":"d052fdfd-3229-47e1-95d0-a3db5599f192","Type":"ContainerDied","Data":"6565a549d0fa0f30f44b056f433c5df5ed77e3d6dd2e31d2ffa5ae29de6e34c9"}
May 25 13:25:41.082222 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.081930    2573 scope.go:117] "RemoveContainer" containerID="8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1"
May 25 13:25:41.082222 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.081871    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"
May 25 13:25:41.091300 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.091274    2573 scope.go:117] "RemoveContainer" containerID="618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f"
May 25 13:25:41.104380 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.104355    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"]
May 25 13:25:41.108616 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.108593    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fvv59j"]
May 25 13:25:41.337800 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.337738    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d052fdfd-3229-47e1-95d0-a3db5599f192" path="/var/lib/kubelet/pods/d052fdfd-3229-47e1-95d0-a3db5599f192/volumes"
May 25 13:25:41.411669 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.411649    2573 scope.go:117] "RemoveContainer" containerID="8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1"
May 25 13:25:41.411952 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:25:41.411933    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1\": container with ID starting with 8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1 not found: ID does not exist" containerID="8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1"
May 25 13:25:41.412012 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.411960    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1"} err="failed to get container status \"8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1\": rpc error: code = NotFound desc = could not find container \"8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1\": container with ID starting with 8cc45a258574fe90bf0e209e3e4f57cdfedc76d7c85540ccff15ae190ca070a1 not found: ID does not exist"
May 25 13:25:41.412012 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.411978    2573 scope.go:117] "RemoveContainer" containerID="618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f"
May 25 13:25:41.412252 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:25:41.412230    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f\": container with ID starting with 618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f not found: ID does not exist" containerID="618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f"
May 25 13:25:41.412319 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:41.412258    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f"} err="failed to get container status \"618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f\": rpc error: code = NotFound desc = could not find container \"618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f\": container with ID starting with 618678f67085214330ea8116c83693ad8b5966cdf8387ca05948b67b226bef5f not found: ID does not exist"
May 25 13:25:50.008964 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.008930    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"]
May 25 13:25:50.009815 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.009779    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d052fdfd-3229-47e1-95d0-a3db5599f192" containerName="storage-initializer"
May 25 13:25:50.010000 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.009986    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d052fdfd-3229-47e1-95d0-a3db5599f192" containerName="storage-initializer"
May 25 13:25:50.010086 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.010075    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d052fdfd-3229-47e1-95d0-a3db5599f192" containerName="main"
May 25 13:25:50.010156 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.010148    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d052fdfd-3229-47e1-95d0-a3db5599f192" containerName="main"
May 25 13:25:50.010500 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.010484    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="d052fdfd-3229-47e1-95d0-a3db5599f192" containerName="main"
May 25 13:25:50.027780 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.027752    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"]
May 25 13:25:50.027908 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.027865    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.030512 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.030475    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:25:50.031496 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.031414    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 13:25:50.031496 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.031430    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
May 25 13:25:50.031496 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.031438    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:25:50.054710 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.054688    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.054825 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.054716    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwcp5\" (UniqueName: \"kubernetes.io/projected/db677a01-315b-4c3e-8108-18a1d57565bc-kube-api-access-zwcp5\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.054825 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.054735    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.054944 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.054862    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db677a01-315b-4c3e-8108-18a1d57565bc-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.054944 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.054912    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.055030 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.054959    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.055030 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.055007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.155826 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.155805    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.156020 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.155835    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.156020 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.155858    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.156020 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.155979    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zwcp5\" (UniqueName: \"kubernetes.io/projected/db677a01-315b-4c3e-8108-18a1d57565bc-kube-api-access-zwcp5\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.156020 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.156019    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.156319 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.156178    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.156319 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.156199    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db677a01-315b-4c3e-8108-18a1d57565bc-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.156319 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.156229    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.156319 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.156247    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.156319 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.156281    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.156542 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.156522    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.158331 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.158308    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.158533 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.158515    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db677a01-315b-4c3e-8108-18a1d57565bc-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.163794 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.163773    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwcp5\" (UniqueName: \"kubernetes.io/projected/db677a01-315b-4c3e-8108-18a1d57565bc-kube-api-access-zwcp5\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.338850 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.338795    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:25:50.465215 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:50.465173    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"]
May 25 13:25:50.467637 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:25:50.467612    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb677a01_315b_4c3e_8108_18a1d57565bc.slice/crio-53f8439de3f67cc851a0ec1378d0fa862397b254f9a06428da1ae6edba150435 WatchSource:0}: Error finding container 53f8439de3f67cc851a0ec1378d0fa862397b254f9a06428da1ae6edba150435: Status 404 returned error can't find the container with id 53f8439de3f67cc851a0ec1378d0fa862397b254f9a06428da1ae6edba150435
May 25 13:25:51.117003 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:51.116957    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq" event={"ID":"db677a01-315b-4c3e-8108-18a1d57565bc","Type":"ContainerStarted","Data":"4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b"}
May 25 13:25:51.117003 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:51.117002    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq" event={"ID":"db677a01-315b-4c3e-8108-18a1d57565bc","Type":"ContainerStarted","Data":"53f8439de3f67cc851a0ec1378d0fa862397b254f9a06428da1ae6edba150435"}
May 25 13:25:53.222958 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:53.222921    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"]
May 25 13:25:53.223386 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:25:53.223212    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq" podUID="db677a01-315b-4c3e-8108-18a1d57565bc" containerName="storage-initializer" containerID="cri-o://4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b" gracePeriod=30
May 25 13:26:18.319622 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.319505    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"]
May 25 13:26:18.322938 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.322911    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.325294 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.325270    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\""
May 25 13:26:18.335583 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.335532    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"]
May 25 13:26:18.351622 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.351597    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.351622 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.351645    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.351622 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.351666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.351923 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.351694    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfbb8\" (UniqueName: \"kubernetes.io/projected/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kube-api-access-bfbb8\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.351923 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.351723    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.351923 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.351758    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.351923 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.351819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.452668 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.452643    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.452769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.452682    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.452769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.452705    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.452769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.452736    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bfbb8\" (UniqueName: \"kubernetes.io/projected/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kube-api-access-bfbb8\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.452927 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.452770    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.452927 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.452815    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.452927 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.452867    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.453122 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.453099    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.453200 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.453176    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.453248 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.453236    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.453349 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.453318    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.455263 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.455239    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.455577 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.455534    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.460906 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.460872    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfbb8\" (UniqueName: \"kubernetes.io/projected/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kube-api-access-bfbb8\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.636636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.636580    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:26:18.764174 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:18.764147    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"]
May 25 13:26:18.766798 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:26:18.766767    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24ecdfa_fce5_44b3_a94d_45a2c2c210f8.slice/crio-320a1bb9b0917ee7157fa9f5dac92d294f69a49e89e3b6f55a15d7ce874ae5c4 WatchSource:0}: Error finding container 320a1bb9b0917ee7157fa9f5dac92d294f69a49e89e3b6f55a15d7ce874ae5c4: Status 404 returned error can't find the container with id 320a1bb9b0917ee7157fa9f5dac92d294f69a49e89e3b6f55a15d7ce874ae5c4
May 25 13:26:19.215919 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:19.215885    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529" event={"ID":"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8","Type":"ContainerStarted","Data":"d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77"}
May 25 13:26:19.216074 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:19.215927    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529" event={"ID":"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8","Type":"ContainerStarted","Data":"320a1bb9b0917ee7157fa9f5dac92d294f69a49e89e3b6f55a15d7ce874ae5c4"}
May 25 13:26:23.414843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.414822    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq_db677a01-315b-4c3e-8108-18a1d57565bc/storage-initializer/0.log"
May 25 13:26:23.415126 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.414887    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:26:23.496844 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.496823    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-dshm\") pod \"db677a01-315b-4c3e-8108-18a1d57565bc\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") "
May 25 13:26:23.496973 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.496854    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-home\") pod \"db677a01-315b-4c3e-8108-18a1d57565bc\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") "
May 25 13:26:23.496973 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.496872    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-kserve-provision-location\") pod \"db677a01-315b-4c3e-8108-18a1d57565bc\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") "
May 25 13:26:23.496973 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.496907    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-model-cache\") pod \"db677a01-315b-4c3e-8108-18a1d57565bc\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") "
May 25 13:26:23.496973 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.496929    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwcp5\" (UniqueName: \"kubernetes.io/projected/db677a01-315b-4c3e-8108-18a1d57565bc-kube-api-access-zwcp5\") pod \"db677a01-315b-4c3e-8108-18a1d57565bc\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") "
May 25 13:26:23.496973 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.496945    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-tmp-dir\") pod \"db677a01-315b-4c3e-8108-18a1d57565bc\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") "
May 25 13:26:23.497228 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.497002    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db677a01-315b-4c3e-8108-18a1d57565bc-tls-certs\") pod \"db677a01-315b-4c3e-8108-18a1d57565bc\" (UID: \"db677a01-315b-4c3e-8108-18a1d57565bc\") "
May 25 13:26:23.497228 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.497118    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-home" (OuterVolumeSpecName: "home") pod "db677a01-315b-4c3e-8108-18a1d57565bc" (UID: "db677a01-315b-4c3e-8108-18a1d57565bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:26:23.497332 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.497256    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "db677a01-315b-4c3e-8108-18a1d57565bc" (UID: "db677a01-315b-4c3e-8108-18a1d57565bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:26:23.497332 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.497294    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db677a01-315b-4c3e-8108-18a1d57565bc" (UID: "db677a01-315b-4c3e-8108-18a1d57565bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:26:23.498958 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.498932    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-dshm" (OuterVolumeSpecName: "dshm") pod "db677a01-315b-4c3e-8108-18a1d57565bc" (UID: "db677a01-315b-4c3e-8108-18a1d57565bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:26:23.499046 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.498989    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db677a01-315b-4c3e-8108-18a1d57565bc-kube-api-access-zwcp5" (OuterVolumeSpecName: "kube-api-access-zwcp5") pod "db677a01-315b-4c3e-8108-18a1d57565bc" (UID: "db677a01-315b-4c3e-8108-18a1d57565bc"). InnerVolumeSpecName "kube-api-access-zwcp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:26:23.499046 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.498991    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db677a01-315b-4c3e-8108-18a1d57565bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db677a01-315b-4c3e-8108-18a1d57565bc" (UID: "db677a01-315b-4c3e-8108-18a1d57565bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:26:23.535311 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.535287    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db677a01-315b-4c3e-8108-18a1d57565bc" (UID: "db677a01-315b-4c3e-8108-18a1d57565bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:26:23.598225 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.598204    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db677a01-315b-4c3e-8108-18a1d57565bc-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:26:23.598225 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.598223    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:26:23.598337 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.598232    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:26:23.598337 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.598241    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:26:23.598337 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.598250    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:26:23.598337 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.598259    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwcp5\" (UniqueName: \"kubernetes.io/projected/db677a01-315b-4c3e-8108-18a1d57565bc-kube-api-access-zwcp5\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:26:23.598337 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:23.598267    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db677a01-315b-4c3e-8108-18a1d57565bc-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:26:24.233976 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:24.233947    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq_db677a01-315b-4c3e-8108-18a1d57565bc/storage-initializer/0.log"
May 25 13:26:24.234156 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:24.233992    2573 generic.go:358] "Generic (PLEG): container finished" podID="db677a01-315b-4c3e-8108-18a1d57565bc" containerID="4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b" exitCode=137
May 25 13:26:24.234156 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:24.234088    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"
May 25 13:26:24.234156 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:24.234095    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq" event={"ID":"db677a01-315b-4c3e-8108-18a1d57565bc","Type":"ContainerDied","Data":"4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b"}
May 25 13:26:24.234156 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:24.234143    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq" event={"ID":"db677a01-315b-4c3e-8108-18a1d57565bc","Type":"ContainerDied","Data":"53f8439de3f67cc851a0ec1378d0fa862397b254f9a06428da1ae6edba150435"}
May 25 13:26:24.234296 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:24.234165    2573 scope.go:117] "RemoveContainer" containerID="4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b"
May 25 13:26:24.253263 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:24.253245    2573 scope.go:117] "RemoveContainer" containerID="4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b"
May 25 13:26:24.253531 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:26:24.253510    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b\": container with ID starting with 4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b not found: ID does not exist" containerID="4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b"
May 25 13:26:24.253581 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:24.253556    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b"} err="failed to get container status \"4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b\": rpc error: code = NotFound desc = could not find container \"4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b\": container with ID starting with 4f1a281be6fd673a92b16840c0606992cadf8e6246275fd69e719590ed490c8b not found: ID does not exist"
May 25 13:26:24.270956 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:24.270931    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"]
May 25 13:26:24.276265 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:24.276239    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f48zvzq"]
May 25 13:26:25.337695 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:26:25.337663    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db677a01-315b-4c3e-8108-18a1d57565bc" path="/var/lib/kubelet/pods/db677a01-315b-4c3e-8108-18a1d57565bc/volumes"
May 25 13:27:37.479359 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:37.479318    2573 generic.go:358] "Generic (PLEG): container finished" podID="b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" containerID="d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77" exitCode=0
May 25 13:27:37.479809 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:37.479390    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529" event={"ID":"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8","Type":"ContainerDied","Data":"d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77"}
May 25 13:27:38.483907 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:38.483866    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529" event={"ID":"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8","Type":"ContainerStarted","Data":"4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf"}
May 25 13:27:38.505725 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:38.505677    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529" podStartSLOduration=80.505663303 podStartE2EDuration="1m20.505663303s" podCreationTimestamp="2026-05-25 13:26:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:27:38.50335013 +0000 UTC m=+1325.764952961" watchObservedRunningTime="2026-05-25 13:27:38.505663303 +0000 UTC m=+1325.767266204"
May 25 13:27:38.637257 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:38.637236    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:27:38.637392 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:38.637366    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:27:38.649714 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:38.649697    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:27:39.498242 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:39.498218    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:27:40.350618 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:40.350585    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"]
May 25 13:27:40.548689 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:27:40.548663    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs: secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 25 13:27:40.549046 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:27:40.548738    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tls-certs podName:b24ecdfa-fce5-44b3-a94d-45a2c2c210f8 nodeName:}" failed. No retries permitted until 2026-05-25 13:27:41.048722399 +0000 UTC m=+1328.310325208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tls-certs") pod "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529" (UID: "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8") : secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 25 13:27:41.053375 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:27:41.053341    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs: secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 25 13:27:41.053539 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:27:41.053415    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tls-certs podName:b24ecdfa-fce5-44b3-a94d-45a2c2c210f8 nodeName:}" failed. No retries permitted until 2026-05-25 13:27:42.053396453 +0000 UTC m=+1329.314999263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tls-certs") pod "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529" (UID: "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8") : secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 25 13:27:41.493242 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.493205    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529" podUID="b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" containerName="main" containerID="cri-o://4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf" gracePeriod=30
May 25 13:27:41.735424 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.735402    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:27:41.859174 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859115    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfbb8\" (UniqueName: \"kubernetes.io/projected/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kube-api-access-bfbb8\") pod \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") "
May 25 13:27:41.859278 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859183    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tmp-dir\") pod \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") "
May 25 13:27:41.859278 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859212    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kserve-provision-location\") pod \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") "
May 25 13:27:41.859278 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859239    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-dshm\") pod \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") "
May 25 13:27:41.859278 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859274    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tls-certs\") pod \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") "
May 25 13:27:41.859447 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859315    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-home\") pod \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") "
May 25 13:27:41.859447 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859342    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-model-cache\") pod \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\" (UID: \"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8\") "
May 25 13:27:41.859576 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859491    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" (UID: "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:27:41.859696 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859639    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:27:41.859696 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859653    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-home" (OuterVolumeSpecName: "home") pod "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" (UID: "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:27:41.859784 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.859727    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" (UID: "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:27:41.861338 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.861309    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kube-api-access-bfbb8" (OuterVolumeSpecName: "kube-api-access-bfbb8") pod "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" (UID: "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8"). InnerVolumeSpecName "kube-api-access-bfbb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:27:41.861809 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.861782    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-dshm" (OuterVolumeSpecName: "dshm") pod "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" (UID: "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:27:41.861911 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.861814    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" (UID: "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:27:41.913106 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.913078    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" (UID: "b24ecdfa-fce5-44b3-a94d-45a2c2c210f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:27:41.960315 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.960290    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:27:41.960315 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.960311    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:27:41.960448 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.960321    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfbb8\" (UniqueName: \"kubernetes.io/projected/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kube-api-access-bfbb8\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:27:41.960448 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.960330    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:27:41.960448 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.960340    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:27:41.960448 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:41.960348    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:27:42.498022 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.497990    2573 generic.go:358] "Generic (PLEG): container finished" podID="b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" containerID="4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf" exitCode=0
May 25 13:27:42.498145 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.498053    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529" event={"ID":"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8","Type":"ContainerDied","Data":"4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf"}
May 25 13:27:42.498145 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.498061    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"
May 25 13:27:42.498145 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.498080    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529" event={"ID":"b24ecdfa-fce5-44b3-a94d-45a2c2c210f8","Type":"ContainerDied","Data":"320a1bb9b0917ee7157fa9f5dac92d294f69a49e89e3b6f55a15d7ce874ae5c4"}
May 25 13:27:42.498145 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.498096    2573 scope.go:117] "RemoveContainer" containerID="4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf"
May 25 13:27:42.507284 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.507267    2573 scope.go:117] "RemoveContainer" containerID="d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77"
May 25 13:27:42.518331 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.518306    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"]
May 25 13:27:42.523083 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.523062    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b9q529"]
May 25 13:27:42.568601 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.568580    2573 scope.go:117] "RemoveContainer" containerID="4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf"
May 25 13:27:42.568848 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:27:42.568829    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf\": container with ID starting with 4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf not found: ID does not exist" containerID="4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf"
May 25 13:27:42.568924 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.568860    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf"} err="failed to get container status \"4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf\": rpc error: code = NotFound desc = could not find container \"4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf\": container with ID starting with 4e27fbc8e94627768aa79b032534e4dd3ccc4494d18750f76cf150739bb81dbf not found: ID does not exist"
May 25 13:27:42.568997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.568885    2573 scope.go:117] "RemoveContainer" containerID="d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77"
May 25 13:27:42.569160 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:27:42.569139    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77\": container with ID starting with d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77 not found: ID does not exist" containerID="d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77"
May 25 13:27:42.569216 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:42.569167    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77"} err="failed to get container status \"d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77\": rpc error: code = NotFound desc = could not find container \"d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77\": container with ID starting with d881fa9c15bc89318585328ee81182aa1466c2d4d127bc5d1ca9cda9d2bf6a77 not found: ID does not exist"
May 25 13:27:43.339219 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:43.339187    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" path="/var/lib/kubelet/pods/b24ecdfa-fce5-44b3-a94d-45a2c2c210f8/volumes"
May 25 13:27:49.735061 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.734980    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"]
May 25 13:27:49.735481 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.735390    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db677a01-315b-4c3e-8108-18a1d57565bc" containerName="storage-initializer"
May 25 13:27:49.735481 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.735408    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="db677a01-315b-4c3e-8108-18a1d57565bc" containerName="storage-initializer"
May 25 13:27:49.735481 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.735426    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" containerName="main"
May 25 13:27:49.735481 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.735434    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" containerName="main"
May 25 13:27:49.735481 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.735454    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" containerName="storage-initializer"
May 25 13:27:49.735481 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.735462    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" containerName="storage-initializer"
May 25 13:27:49.735797 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.735594    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="db677a01-315b-4c3e-8108-18a1d57565bc" containerName="storage-initializer"
May 25 13:27:49.735797 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.735612    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="b24ecdfa-fce5-44b3-a94d-45a2c2c210f8" containerName="main"
May 25 13:27:49.739157 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.739136    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.741520 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.741499    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:27:49.741643 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.741529    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 25 13:27:49.741643 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.741617    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 13:27:49.741762 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.741658    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:27:49.753260 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.753233    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"]
May 25 13:27:49.816109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.816078    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4xm\" (UniqueName: \"kubernetes.io/projected/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kube-api-access-wf4xm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.816219 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.816117    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.816219 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.816147    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.816290 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.816226    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.816290 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.816255    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.816290 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.816280    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.816386 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.816356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.917407 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.917375    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.917407 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.917414    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.917540 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.917442    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.917540 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.917490    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.917540 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.917533    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4xm\" (UniqueName: \"kubernetes.io/projected/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kube-api-access-wf4xm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.917709 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.917672    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.917767 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.917718    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.917846 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.917825    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.917920 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.917845    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.917975 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.917964    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.918057 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.918041    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.919800 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.919777    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.920069 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.920050    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:49.925301 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:49.925272    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4xm\" (UniqueName: \"kubernetes.io/projected/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kube-api-access-wf4xm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:50.051901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:50.051876    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:27:50.177930 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:50.177907    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"]
May 25 13:27:50.180461 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:27:50.180431    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4e0e37_a115_4fbd_9d69_0ed360487a6b.slice/crio-b932e21b3fd5191788777180c2ae994bdb03143daab5c28b98687c706af47de5 WatchSource:0}: Error finding container b932e21b3fd5191788777180c2ae994bdb03143daab5c28b98687c706af47de5: Status 404 returned error can't find the container with id b932e21b3fd5191788777180c2ae994bdb03143daab5c28b98687c706af47de5
May 25 13:27:50.525177 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:50.525145    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" event={"ID":"7d4e0e37-a115-4fbd-9d69-0ed360487a6b","Type":"ContainerStarted","Data":"ac9c36faf40577750c6bde63a9eba8fcf19a69dcf8af5822e5196e0fc6dc6484"}
May 25 13:27:50.525177 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:27:50.525180    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" event={"ID":"7d4e0e37-a115-4fbd-9d69-0ed360487a6b","Type":"ContainerStarted","Data":"b932e21b3fd5191788777180c2ae994bdb03143daab5c28b98687c706af47de5"}
May 25 13:28:46.704565 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:28:46.704519    2573 generic.go:358] "Generic (PLEG): container finished" podID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerID="ac9c36faf40577750c6bde63a9eba8fcf19a69dcf8af5822e5196e0fc6dc6484" exitCode=0
May 25 13:28:46.704996 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:28:46.704599    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" event={"ID":"7d4e0e37-a115-4fbd-9d69-0ed360487a6b","Type":"ContainerDied","Data":"ac9c36faf40577750c6bde63a9eba8fcf19a69dcf8af5822e5196e0fc6dc6484"}
May 25 13:29:22.838801 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:29:22.838762    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" event={"ID":"7d4e0e37-a115-4fbd-9d69-0ed360487a6b","Type":"ContainerStarted","Data":"a43bf0d0f0be88d73484aa0d61cbc0286c0e3edf53c6b087f1ece94aa50b73d1"}
May 25 13:29:22.864075 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:29:22.864019    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" podStartSLOduration=58.599456585 podStartE2EDuration="1m33.864001568s" podCreationTimestamp="2026-05-25 13:27:49 +0000 UTC" firstStartedPulling="2026-05-25 13:28:46.705747103 +0000 UTC m=+1393.967349912" lastFinishedPulling="2026-05-25 13:29:21.970292083 +0000 UTC m=+1429.231894895" observedRunningTime="2026-05-25 13:29:22.862081911 +0000 UTC m=+1430.123684742" watchObservedRunningTime="2026-05-25 13:29:22.864001568 +0000 UTC m=+1430.125604410"
May 25 13:29:30.052196 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:29:30.052151    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:29:30.052782 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:29:30.052244    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:29:30.053759 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:29:30.053734    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
May 25 13:29:40.052916 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:29:40.052868    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
May 25 13:29:50.052996 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:29:50.052957    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
May 25 13:30:00.052868 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:30:00.052826    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
May 25 13:30:10.053222 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:30:10.053176    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
May 25 13:30:20.052457 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:30:20.052414    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
May 25 13:30:30.052286 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:30:30.052236    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
May 25 13:30:33.342828 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:30:33.342804    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:30:33.344013 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:30:33.343990    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:30:40.053080 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:30:40.053037    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
May 25 13:30:50.062811 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:30:50.062729    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:30:50.070376 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:30:50.070347    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:31:02.244483 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.244451    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"]
May 25 13:31:02.247650 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.247630    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.250002 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.249981    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
May 25 13:31:02.261390 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.261369    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"]
May 25 13:31:02.391519 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.391496    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tls-certs\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.391625 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.391532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.391625 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.391603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tmp-dir\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.391701 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.391639    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-dshm\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.391701 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.391671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-home\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.391701 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.391688    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf2gq\" (UniqueName: \"kubernetes.io/projected/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kube-api-access-wf2gq\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.391792 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.391744    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-model-cache\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492230 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492207    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tls-certs\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492321 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492241    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492321 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492266    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tmp-dir\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492321 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492283    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-dshm\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492321 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492307    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-home\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492590 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492327    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wf2gq\" (UniqueName: \"kubernetes.io/projected/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kube-api-access-wf2gq\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492590 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492366    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-model-cache\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492708 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492601    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492708 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492652    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tmp-dir\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492806 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492790    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-model-cache\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.492845 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.492829    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-home\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.494559 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.494496    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-dshm\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.494687 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.494672    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tls-certs\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.500971 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.500952    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf2gq\" (UniqueName: \"kubernetes.io/projected/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kube-api-access-wf2gq\") pod \"precise-prefix-cache-test-kserve-84f494576-gk56c\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.557037 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.557018    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:31:02.684601 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.684574    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"]
May 25 13:31:02.685763 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:31:02.685738    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2df3ec_9f0f_4734_acad_c0a9b086bed7.slice/crio-9d89b9e8a7041f771825fcc6f36437bfece21960dee92fdc3b9f3b5322d10cd0 WatchSource:0}: Error finding container 9d89b9e8a7041f771825fcc6f36437bfece21960dee92fdc3b9f3b5322d10cd0: Status 404 returned error can't find the container with id 9d89b9e8a7041f771825fcc6f36437bfece21960dee92fdc3b9f3b5322d10cd0
May 25 13:31:02.687540 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:02.687523    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:31:03.162093 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:03.162057    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c" event={"ID":"1c2df3ec-9f0f-4734-acad-c0a9b086bed7","Type":"ContainerStarted","Data":"51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a"}
May 25 13:31:03.162093 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:03.162095    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c" event={"ID":"1c2df3ec-9f0f-4734-acad-c0a9b086bed7","Type":"ContainerStarted","Data":"9d89b9e8a7041f771825fcc6f36437bfece21960dee92fdc3b9f3b5322d10cd0"}
May 25 13:31:15.067434 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:15.067398    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"]
May 25 13:31:15.067884 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:15.067787    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main" containerID="cri-o://a43bf0d0f0be88d73484aa0d61cbc0286c0e3edf53c6b087f1ece94aa50b73d1" gracePeriod=30
May 25 13:31:23.218505 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.218468    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"]
May 25 13:31:23.225732 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.225708    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.228878 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.228849    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 25 13:31:23.241632 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.241606    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"]
May 25 13:31:23.255664 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.255636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-tmp-dir\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.255773 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.255665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdll\" (UniqueName: \"kubernetes.io/projected/bed59595-671d-4abd-8e90-16bdb6231ded-kube-api-access-6pdll\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.255773 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.255699    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed59595-671d-4abd-8e90-16bdb6231ded-tls-certs\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.255773 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.255761    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-home\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.255886 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.255800    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-dshm\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.255886 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.255819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-model-cache\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.255886 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.255874    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356371 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356347    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-tmp-dir\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356475 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356376    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdll\" (UniqueName: \"kubernetes.io/projected/bed59595-671d-4abd-8e90-16bdb6231ded-kube-api-access-6pdll\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356475 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356405    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed59595-671d-4abd-8e90-16bdb6231ded-tls-certs\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356475 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356427    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-home\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356475 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356457    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-dshm\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356475 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356474    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-model-cache\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356775 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356506    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356775 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356759    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-tmp-dir\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356891 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356813    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356891 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356857    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-model-cache\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.356996 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.356924    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-home\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.358870 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.358837    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-dshm\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.358974 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.358894    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed59595-671d-4abd-8e90-16bdb6231ded-tls-certs\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.364491 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.364462    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdll\" (UniqueName: \"kubernetes.io/projected/bed59595-671d-4abd-8e90-16bdb6231ded-kube-api-access-6pdll\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5bxq5\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.537153 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.537132    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:31:23.667269 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:23.667241    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"]
May 25 13:31:23.669658 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:31:23.669627    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed59595_671d_4abd_8e90_16bdb6231ded.slice/crio-8a9251164f0f9b96d455c31683c06b44804ddf2782b6ee79b2d5bec14a4d1278 WatchSource:0}: Error finding container 8a9251164f0f9b96d455c31683c06b44804ddf2782b6ee79b2d5bec14a4d1278: Status 404 returned error can't find the container with id 8a9251164f0f9b96d455c31683c06b44804ddf2782b6ee79b2d5bec14a4d1278
May 25 13:31:24.230941 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:24.230906    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" event={"ID":"bed59595-671d-4abd-8e90-16bdb6231ded","Type":"ContainerStarted","Data":"266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb"}
May 25 13:31:24.230941 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:24.230945    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" event={"ID":"bed59595-671d-4abd-8e90-16bdb6231ded","Type":"ContainerStarted","Data":"8a9251164f0f9b96d455c31683c06b44804ddf2782b6ee79b2d5bec14a4d1278"}
May 25 13:31:45.307277 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.307251    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6_7d4e0e37-a115-4fbd-9d69-0ed360487a6b/main/0.log"
May 25 13:31:45.307678 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.307640    2573 generic.go:358] "Generic (PLEG): container finished" podID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerID="a43bf0d0f0be88d73484aa0d61cbc0286c0e3edf53c6b087f1ece94aa50b73d1" exitCode=137
May 25 13:31:45.308792 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.307714    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" event={"ID":"7d4e0e37-a115-4fbd-9d69-0ed360487a6b","Type":"ContainerDied","Data":"a43bf0d0f0be88d73484aa0d61cbc0286c0e3edf53c6b087f1ece94aa50b73d1"}
May 25 13:31:45.331769 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.331720    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6_7d4e0e37-a115-4fbd-9d69-0ed360487a6b/main/0.log"
May 25 13:31:45.332062 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.332046    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:31:45.426786 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.426762    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kserve-provision-location\") pod \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") "
May 25 13:31:45.426901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.426791    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-dshm\") pod \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") "
May 25 13:31:45.426901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.426811    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tls-certs\") pod \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") "
May 25 13:31:45.426901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.426879    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tmp-dir\") pod \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") "
May 25 13:31:45.427033 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.426920    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-home\") pod \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") "
May 25 13:31:45.427033 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.426990    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-model-cache\") pod \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") "
May 25 13:31:45.427033 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.427019    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf4xm\" (UniqueName: \"kubernetes.io/projected/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kube-api-access-wf4xm\") pod \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\" (UID: \"7d4e0e37-a115-4fbd-9d69-0ed360487a6b\") "
May 25 13:31:45.427528 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.427493    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d4e0e37-a115-4fbd-9d69-0ed360487a6b" (UID: "7d4e0e37-a115-4fbd-9d69-0ed360487a6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:31:45.427720 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.427687    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-home" (OuterVolumeSpecName: "home") pod "7d4e0e37-a115-4fbd-9d69-0ed360487a6b" (UID: "7d4e0e37-a115-4fbd-9d69-0ed360487a6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:31:45.429076 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.429046    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d4e0e37-a115-4fbd-9d69-0ed360487a6b" (UID: "7d4e0e37-a115-4fbd-9d69-0ed360487a6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:31:45.429182 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.429136    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kube-api-access-wf4xm" (OuterVolumeSpecName: "kube-api-access-wf4xm") pod "7d4e0e37-a115-4fbd-9d69-0ed360487a6b" (UID: "7d4e0e37-a115-4fbd-9d69-0ed360487a6b"). InnerVolumeSpecName "kube-api-access-wf4xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:31:45.429182 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.429145    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-dshm" (OuterVolumeSpecName: "dshm") pod "7d4e0e37-a115-4fbd-9d69-0ed360487a6b" (UID: "7d4e0e37-a115-4fbd-9d69-0ed360487a6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:31:45.439729 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.439701    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d4e0e37-a115-4fbd-9d69-0ed360487a6b" (UID: "7d4e0e37-a115-4fbd-9d69-0ed360487a6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:31:45.483847 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.483817    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d4e0e37-a115-4fbd-9d69-0ed360487a6b" (UID: "7d4e0e37-a115-4fbd-9d69-0ed360487a6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:31:45.528282 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.528259    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:31:45.528282 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.528281    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:31:45.528405 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.528292    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:31:45.528405 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.528301    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:31:45.528405 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.528309    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:31:45.528405 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.528318    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:31:45.528405 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:45.528326    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4xm\" (UniqueName: \"kubernetes.io/projected/7d4e0e37-a115-4fbd-9d69-0ed360487a6b-kube-api-access-wf4xm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:31:46.313391 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:46.313361    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6_7d4e0e37-a115-4fbd-9d69-0ed360487a6b/main/0.log"
May 25 13:31:46.313856 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:46.313828    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6" event={"ID":"7d4e0e37-a115-4fbd-9d69-0ed360487a6b","Type":"ContainerDied","Data":"b932e21b3fd5191788777180c2ae994bdb03143daab5c28b98687c706af47de5"}
May 25 13:31:46.313923 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:46.313858    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"
May 25 13:31:46.313923 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:46.313877    2573 scope.go:117] "RemoveContainer" containerID="a43bf0d0f0be88d73484aa0d61cbc0286c0e3edf53c6b087f1ece94aa50b73d1"
May 25 13:31:46.323355 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:46.323339    2573 scope.go:117] "RemoveContainer" containerID="ac9c36faf40577750c6bde63a9eba8fcf19a69dcf8af5822e5196e0fc6dc6484"
May 25 13:31:46.338286 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:46.338264    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"]
May 25 13:31:46.343735 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:46.343709    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-sgts6"]
May 25 13:31:47.337947 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:31:47.337916    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" path="/var/lib/kubelet/pods/7d4e0e37-a115-4fbd-9d69-0ed360487a6b/volumes"
May 25 13:34:31.852802 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:34:31.852766    2573 generic.go:358] "Generic (PLEG): container finished" podID="bed59595-671d-4abd-8e90-16bdb6231ded" containerID="266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb" exitCode=0
May 25 13:34:31.853224 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:34:31.852843    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" event={"ID":"bed59595-671d-4abd-8e90-16bdb6231ded","Type":"ContainerDied","Data":"266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb"}
May 25 13:34:32.858727 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:34:32.858697    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" event={"ID":"bed59595-671d-4abd-8e90-16bdb6231ded","Type":"ContainerStarted","Data":"aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d"}
May 25 13:34:32.881265 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:34:32.881200    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podStartSLOduration=189.881186417 podStartE2EDuration="3m9.881186417s" podCreationTimestamp="2026-05-25 13:31:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:34:32.879686561 +0000 UTC m=+1740.141289394" watchObservedRunningTime="2026-05-25 13:34:32.881186417 +0000 UTC m=+1740.142789248"
May 25 13:34:33.538223 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:34:33.538194    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:34:33.538367 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:34:33.538334    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:34:33.539718 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:34:33.539688    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 25 13:34:43.538305 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:34:43.538265    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 25 13:34:53.537935 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:34:53.537883    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 25 13:35:03.537744 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:35:03.537700    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 25 13:35:13.538525 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:35:13.538486    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 25 13:35:23.538336 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:35:23.538281    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 25 13:35:33.364834 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:35:33.364799    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:35:33.367328 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:35:33.367297    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:35:33.538566 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:35:33.538511    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 25 13:35:43.537704 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:35:43.537662    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 25 13:35:53.537759 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:35:53.537711    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 25 13:36:03.548295 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:03.548254    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:36:03.556332 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:03.556309    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:36:09.476118 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:09.476082    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"]
May 25 13:36:09.476509 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:09.476424    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main" containerID="cri-o://aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d" gracePeriod=30
May 25 13:36:19.726179 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.726141    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"]
May 25 13:36:19.726762 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.726635    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="storage-initializer"
May 25 13:36:19.726762 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.726656    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="storage-initializer"
May 25 13:36:19.726762 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.726687    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main"
May 25 13:36:19.726762 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.726696    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main"
May 25 13:36:19.726979 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.726776    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="7d4e0e37-a115-4fbd-9d69-0ed360487a6b" containerName="main"
May 25 13:36:19.731332 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.730997    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.735958 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.735940    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 25 13:36:19.746618 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.746596    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"]
May 25 13:36:19.791405 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.791385    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqhw8\" (UniqueName: \"kubernetes.io/projected/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kube-api-access-cqhw8\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.791516 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.791415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-model-cache\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.791516 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.791440    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tls-certs\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.791516 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.791469    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kserve-provision-location\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.791516 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.791490    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tmp-dir\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.791740 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.791521    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-home\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.791740 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.791562    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-dshm\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892121    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-home\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892239 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892148    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-dshm\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892239 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892189    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cqhw8\" (UniqueName: \"kubernetes.io/projected/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kube-api-access-cqhw8\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892344 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892314    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-model-cache\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892407 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892377    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tls-certs\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892468 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892419    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kserve-provision-location\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892468 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892450    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tmp-dir\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892658 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892482    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-home\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892723 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892657    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-model-cache\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892778 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892730    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kserve-provision-location\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.892876 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.892847    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tmp-dir\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.894673 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.894646    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-dshm\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.894804 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.894788    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tls-certs\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:19.900094 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:19.900065    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqhw8\" (UniqueName: \"kubernetes.io/projected/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kube-api-access-cqhw8\") pod \"router-with-refs-test-kserve-76745cd4f8-lhchc\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:20.042162 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:20.042137    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:36:20.171590 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:20.171537    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"]
May 25 13:36:20.174442 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:36:20.174413    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bb2522_ddd4_4fbc_ba37_68fb67c3d072.slice/crio-d8817edfefa1107cd98ec3aa04556037fe5beccdac6e41a11ce54076de9f3bfe WatchSource:0}: Error finding container d8817edfefa1107cd98ec3aa04556037fe5beccdac6e41a11ce54076de9f3bfe: Status 404 returned error can't find the container with id d8817edfefa1107cd98ec3aa04556037fe5beccdac6e41a11ce54076de9f3bfe
May 25 13:36:20.176426 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:20.176404    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:36:20.218852 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:20.218826    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" event={"ID":"44bb2522-ddd4-4fbc-ba37-68fb67c3d072","Type":"ContainerStarted","Data":"d8817edfefa1107cd98ec3aa04556037fe5beccdac6e41a11ce54076de9f3bfe"}
May 25 13:36:21.225517 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:21.225485    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" event={"ID":"44bb2522-ddd4-4fbc-ba37-68fb67c3d072","Type":"ContainerStarted","Data":"e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68"}
May 25 13:36:39.687418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.687396    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-65d8bf8b78-5bxq5_bed59595-671d-4abd-8e90-16bdb6231ded/main/0.log"
May 25 13:36:39.687756 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.687742    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:36:39.737489 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.737467    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-tmp-dir\") pod \"bed59595-671d-4abd-8e90-16bdb6231ded\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") "
May 25 13:36:39.737620 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.737497    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed59595-671d-4abd-8e90-16bdb6231ded-tls-certs\") pod \"bed59595-671d-4abd-8e90-16bdb6231ded\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") "
May 25 13:36:39.737620 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.737513    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-model-cache\") pod \"bed59595-671d-4abd-8e90-16bdb6231ded\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") "
May 25 13:36:39.737620 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.737566    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pdll\" (UniqueName: \"kubernetes.io/projected/bed59595-671d-4abd-8e90-16bdb6231ded-kube-api-access-6pdll\") pod \"bed59595-671d-4abd-8e90-16bdb6231ded\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") "
May 25 13:36:39.737620 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.737589    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-home\") pod \"bed59595-671d-4abd-8e90-16bdb6231ded\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") "
May 25 13:36:39.737829 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.737636    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-dshm\") pod \"bed59595-671d-4abd-8e90-16bdb6231ded\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") "
May 25 13:36:39.737829 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.737664    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-kserve-provision-location\") pod \"bed59595-671d-4abd-8e90-16bdb6231ded\" (UID: \"bed59595-671d-4abd-8e90-16bdb6231ded\") "
May 25 13:36:39.737829 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.737787    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-model-cache" (OuterVolumeSpecName: "model-cache") pod "bed59595-671d-4abd-8e90-16bdb6231ded" (UID: "bed59595-671d-4abd-8e90-16bdb6231ded"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:36:39.737997 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.737952    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:36:39.738243 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.738208    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-home" (OuterVolumeSpecName: "home") pod "bed59595-671d-4abd-8e90-16bdb6231ded" (UID: "bed59595-671d-4abd-8e90-16bdb6231ded"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:36:39.739738 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.739710    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed59595-671d-4abd-8e90-16bdb6231ded-kube-api-access-6pdll" (OuterVolumeSpecName: "kube-api-access-6pdll") pod "bed59595-671d-4abd-8e90-16bdb6231ded" (UID: "bed59595-671d-4abd-8e90-16bdb6231ded"). InnerVolumeSpecName "kube-api-access-6pdll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:36:39.739915 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.739891    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed59595-671d-4abd-8e90-16bdb6231ded-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bed59595-671d-4abd-8e90-16bdb6231ded" (UID: "bed59595-671d-4abd-8e90-16bdb6231ded"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:36:39.740024 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.740004    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-dshm" (OuterVolumeSpecName: "dshm") pod "bed59595-671d-4abd-8e90-16bdb6231ded" (UID: "bed59595-671d-4abd-8e90-16bdb6231ded"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:36:39.750117 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.750093    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bed59595-671d-4abd-8e90-16bdb6231ded" (UID: "bed59595-671d-4abd-8e90-16bdb6231ded"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:36:39.798116 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.798093    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bed59595-671d-4abd-8e90-16bdb6231ded" (UID: "bed59595-671d-4abd-8e90-16bdb6231ded"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:36:39.838538 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.838517    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:36:39.838619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.838537    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:36:39.838619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.838565    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:36:39.838619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.838575    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed59595-671d-4abd-8e90-16bdb6231ded-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:36:39.838619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.838583    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pdll\" (UniqueName: \"kubernetes.io/projected/bed59595-671d-4abd-8e90-16bdb6231ded-kube-api-access-6pdll\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:36:39.838619 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:39.838591    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed59595-671d-4abd-8e90-16bdb6231ded-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:36:40.288604 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.288581    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-65d8bf8b78-5bxq5_bed59595-671d-4abd-8e90-16bdb6231ded/main/0.log"
May 25 13:36:40.288982 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.288955    2573 generic.go:358] "Generic (PLEG): container finished" podID="bed59595-671d-4abd-8e90-16bdb6231ded" containerID="aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d" exitCode=137
May 25 13:36:40.289074 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.289026    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"
May 25 13:36:40.289074 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.289039    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" event={"ID":"bed59595-671d-4abd-8e90-16bdb6231ded","Type":"ContainerDied","Data":"aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d"}
May 25 13:36:40.289156 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.289081    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5" event={"ID":"bed59595-671d-4abd-8e90-16bdb6231ded","Type":"ContainerDied","Data":"8a9251164f0f9b96d455c31683c06b44804ddf2782b6ee79b2d5bec14a4d1278"}
May 25 13:36:40.289156 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.289098    2573 scope.go:117] "RemoveContainer" containerID="aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d"
May 25 13:36:40.298400 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.298382    2573 scope.go:117] "RemoveContainer" containerID="266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb"
May 25 13:36:40.308167 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.308149    2573 scope.go:117] "RemoveContainer" containerID="aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d"
May 25 13:36:40.308434 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:36:40.308411    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d\": container with ID starting with aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d not found: ID does not exist" containerID="aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d"
May 25 13:36:40.308473 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.308445    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d"} err="failed to get container status \"aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d\": rpc error: code = NotFound desc = could not find container \"aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d\": container with ID starting with aaad48b9da7e58d39b60d83a7d07ef05ea4aebb30ccc4ce665ad95681141434d not found: ID does not exist"
May 25 13:36:40.308473 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.308464    2573 scope.go:117] "RemoveContainer" containerID="266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb"
May 25 13:36:40.308731 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:36:40.308711    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb\": container with ID starting with 266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb not found: ID does not exist" containerID="266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb"
May 25 13:36:40.308787 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.308737    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb"} err="failed to get container status \"266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb\": rpc error: code = NotFound desc = could not find container \"266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb\": container with ID starting with 266ee553ec0f5dcaf65bf69fba7590b1f387e076e7964fc73b065cd27cd9bcdb not found: ID does not exist"
May 25 13:36:40.313081 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.313057    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"]
May 25 13:36:40.328922 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:40.328900    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5bxq5"]
May 25 13:36:41.337888 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:36:41.337854    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" path="/var/lib/kubelet/pods/bed59595-671d-4abd-8e90-16bdb6231ded/volumes"
May 25 13:38:07.572311 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:07.572234    2573 generic.go:358] "Generic (PLEG): container finished" podID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerID="e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68" exitCode=0
May 25 13:38:07.572311 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:07.572275    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" event={"ID":"44bb2522-ddd4-4fbc-ba37-68fb67c3d072","Type":"ContainerDied","Data":"e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68"}
May 25 13:38:08.576930 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:08.576895    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" event={"ID":"44bb2522-ddd4-4fbc-ba37-68fb67c3d072","Type":"ContainerStarted","Data":"f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef"}
May 25 13:38:08.599806 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:08.599748    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podStartSLOduration=109.599728929 podStartE2EDuration="1m49.599728929s" podCreationTimestamp="2026-05-25 13:36:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:38:08.598027638 +0000 UTC m=+1955.859630469" watchObservedRunningTime="2026-05-25 13:38:08.599728929 +0000 UTC m=+1955.861331765"
May 25 13:38:10.043023 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:10.042989    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:38:10.043023 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:10.043026    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:38:10.044392 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:10.044365    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 25 13:38:20.043289 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:20.043197    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 25 13:38:30.042831 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:30.042786    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 25 13:38:40.042884 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:40.042831    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 25 13:38:50.042778 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:38:50.042731    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 25 13:39:00.042697 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:00.042651    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 25 13:39:10.042976 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:10.042938    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 25 13:39:20.043369 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:20.043328    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 25 13:39:30.042838 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:30.042793    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 25 13:39:40.052443 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:40.052407    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:39:40.060224 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:40.060195    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:39:45.541175 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:45.541100    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"]
May 25 13:39:45.541663 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:45.541426    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main" containerID="cri-o://f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef" gracePeriod=30
May 25 13:39:55.734571 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.730964    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"]
May 25 13:39:55.734571 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.731768    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="storage-initializer"
May 25 13:39:55.734571 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.731802    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="storage-initializer"
May 25 13:39:55.734571 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.731822    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main"
May 25 13:39:55.734571 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.731833    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main"
May 25 13:39:55.734571 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.731991    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="bed59595-671d-4abd-8e90-16bdb6231ded" containerName="main"
May 25 13:39:55.738133 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.738098    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.741047 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.740633    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 25 13:39:55.741047 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.740733    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-lkjvf\""
May 25 13:39:55.741875 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.741846    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"]
May 25 13:39:55.746408 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.746385    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"]
May 25 13:39:55.746580 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.746555    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.764494 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.764470    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"]
May 25 13:39:55.851349 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.851448 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.851448 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851374    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.851448 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5lfs\" (UniqueName: \"kubernetes.io/projected/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kube-api-access-x5lfs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.851570 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.851570 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851490    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.851570 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851510    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.851570 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.851704 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851573    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jttbs\" (UniqueName: \"kubernetes.io/projected/686f5ed7-e758-489f-aa12-0eeb4c756636-kube-api-access-jttbs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.851704 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851595    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.851704 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686f5ed7-e758-489f-aa12-0eeb4c756636-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.851704 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.851704 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851680    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.851704 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.851702    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.952762 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.952741    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.952880 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.952780    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.952880 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.952806    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.952880 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.952833    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x5lfs\" (UniqueName: \"kubernetes.io/projected/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kube-api-access-x5lfs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.952880 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.952859    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.953096 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.952891    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.953096 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.952920    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.953096 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.952954    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.953096 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.952984    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jttbs\" (UniqueName: \"kubernetes.io/projected/686f5ed7-e758-489f-aa12-0eeb4c756636-kube-api-access-jttbs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.953096 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953018    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.953096 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953066    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686f5ed7-e758-489f-aa12-0eeb4c756636-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.953383 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953095    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.953383 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953125    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.953383 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953168    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.953383 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953173    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.953383 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953339    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.953383 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953349    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.953725 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953449    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.953725 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953505    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.953725 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953523    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.953883 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953761    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.953942 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.953905    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.955250 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.955227    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.955629 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.955608    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686f5ed7-e758-489f-aa12-0eeb4c756636-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.955690 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.955665    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.955877 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.955859    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:55.961309 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.961283    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jttbs\" (UniqueName: \"kubernetes.io/projected/686f5ed7-e758-489f-aa12-0eeb4c756636-kube-api-access-jttbs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:55.961395 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:55.961378    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5lfs\" (UniqueName: \"kubernetes.io/projected/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kube-api-access-x5lfs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:56.054409 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:56.054384    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:56.060915 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:56.060883    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:39:56.198907 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:56.198792    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"]
May 25 13:39:56.201141 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:39:56.201113    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb486f451_c7a6_4d65_906e_1ffe3ce69b14.slice/crio-c30ca3bdacc1746a3f5c76b205aed477249b4c40d62e89f7944d80ae5cb4e913 WatchSource:0}: Error finding container c30ca3bdacc1746a3f5c76b205aed477249b4c40d62e89f7944d80ae5cb4e913: Status 404 returned error can't find the container with id c30ca3bdacc1746a3f5c76b205aed477249b4c40d62e89f7944d80ae5cb4e913
May 25 13:39:56.223388 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:56.223367    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"]
May 25 13:39:56.225349 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:39:56.225326    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686f5ed7_e758_489f_aa12_0eeb4c756636.slice/crio-51ed1f57cad4a9df59c1b560c9a40e2212cd5f011d5317ce5a350b55b09b0fbf WatchSource:0}: Error finding container 51ed1f57cad4a9df59c1b560c9a40e2212cd5f011d5317ce5a350b55b09b0fbf: Status 404 returned error can't find the container with id 51ed1f57cad4a9df59c1b560c9a40e2212cd5f011d5317ce5a350b55b09b0fbf
May 25 13:39:56.945618 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:56.945578    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" event={"ID":"686f5ed7-e758-489f-aa12-0eeb4c756636","Type":"ContainerStarted","Data":"d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020"}
May 25 13:39:56.945618 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:56.945624    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" event={"ID":"686f5ed7-e758-489f-aa12-0eeb4c756636","Type":"ContainerStarted","Data":"51ed1f57cad4a9df59c1b560c9a40e2212cd5f011d5317ce5a350b55b09b0fbf"}
May 25 13:39:56.946695 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:56.946664    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" event={"ID":"b486f451-c7a6-4d65-906e-1ffe3ce69b14","Type":"ContainerStarted","Data":"c30ca3bdacc1746a3f5c76b205aed477249b4c40d62e89f7944d80ae5cb4e913"}
May 25 13:39:57.955067 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:57.955023    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" event={"ID":"b486f451-c7a6-4d65-906e-1ffe3ce69b14","Type":"ContainerStarted","Data":"2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5"}
May 25 13:39:57.955527 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:57.955130    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:39:58.960981 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:39:58.960929    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" event={"ID":"b486f451-c7a6-4d65-906e-1ffe3ce69b14","Type":"ContainerStarted","Data":"cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150"}
May 25 13:40:01.973105 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:01.973072    2573 generic.go:358] "Generic (PLEG): container finished" podID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerID="cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150" exitCode=0
May 25 13:40:01.973438 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:01.973114    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" event={"ID":"b486f451-c7a6-4d65-906e-1ffe3ce69b14","Type":"ContainerDied","Data":"cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150"}
May 25 13:40:02.978618 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:02.978585    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" event={"ID":"b486f451-c7a6-4d65-906e-1ffe3ce69b14","Type":"ContainerStarted","Data":"8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061"}
May 25 13:40:03.002167 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:03.002111    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podStartSLOduration=7.172530299 podStartE2EDuration="8.002094036s" podCreationTimestamp="2026-05-25 13:39:55 +0000 UTC" firstStartedPulling="2026-05-25 13:39:56.202939709 +0000 UTC m=+2063.464542517" lastFinishedPulling="2026-05-25 13:39:57.032503445 +0000 UTC m=+2064.294106254" observedRunningTime="2026-05-25 13:40:03.000589425 +0000 UTC m=+2070.262192256" watchObservedRunningTime="2026-05-25 13:40:03.002094036 +0000 UTC m=+2070.263696868"
May 25 13:40:06.054937 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:06.054893    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:40:06.054937 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:06.054936    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:40:06.056138 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:06.056106    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8001/health\": dial tcp 10.132.0.33:8001: connect: connection refused"
May 25 13:40:15.820539 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.820509    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-76745cd4f8-lhchc_44bb2522-ddd4-4fbc-ba37-68fb67c3d072/main/0.log"
May 25 13:40:15.820956 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.820938    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:40:15.918737 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.918708    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-dshm\") pod \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") "
May 25 13:40:15.918872 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.918745    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kserve-provision-location\") pod \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") "
May 25 13:40:15.918872 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.918769    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqhw8\" (UniqueName: \"kubernetes.io/projected/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kube-api-access-cqhw8\") pod \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") "
May 25 13:40:15.918872 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.918798    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-model-cache\") pod \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") "
May 25 13:40:15.918872 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.918826    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tls-certs\") pod \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") "
May 25 13:40:15.918872 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.918844    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-home\") pod \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") "
May 25 13:40:15.919130 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.918888    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tmp-dir\") pod \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\" (UID: \"44bb2522-ddd4-4fbc-ba37-68fb67c3d072\") "
May 25 13:40:15.919130 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.919028    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-model-cache" (OuterVolumeSpecName: "model-cache") pod "44bb2522-ddd4-4fbc-ba37-68fb67c3d072" (UID: "44bb2522-ddd4-4fbc-ba37-68fb67c3d072"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:40:15.919237 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.919178    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:40:15.919590 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.919530    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-home" (OuterVolumeSpecName: "home") pod "44bb2522-ddd4-4fbc-ba37-68fb67c3d072" (UID: "44bb2522-ddd4-4fbc-ba37-68fb67c3d072"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:40:15.921140 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.921111    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kube-api-access-cqhw8" (OuterVolumeSpecName: "kube-api-access-cqhw8") pod "44bb2522-ddd4-4fbc-ba37-68fb67c3d072" (UID: "44bb2522-ddd4-4fbc-ba37-68fb67c3d072"). InnerVolumeSpecName "kube-api-access-cqhw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:40:15.921381 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.921355    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44bb2522-ddd4-4fbc-ba37-68fb67c3d072" (UID: "44bb2522-ddd4-4fbc-ba37-68fb67c3d072"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:40:15.921381 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.921369    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-dshm" (OuterVolumeSpecName: "dshm") pod "44bb2522-ddd4-4fbc-ba37-68fb67c3d072" (UID: "44bb2522-ddd4-4fbc-ba37-68fb67c3d072"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:40:15.939457 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.939431    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44bb2522-ddd4-4fbc-ba37-68fb67c3d072" (UID: "44bb2522-ddd4-4fbc-ba37-68fb67c3d072"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:40:15.989739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:15.989707    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44bb2522-ddd4-4fbc-ba37-68fb67c3d072" (UID: "44bb2522-ddd4-4fbc-ba37-68fb67c3d072"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:40:16.020066 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.020040    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:40:16.020066 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.020064    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:40:16.020210 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.020078    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:40:16.020210 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.020090    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:40:16.020210 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.020102    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:40:16.020210 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.020115    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqhw8\" (UniqueName: \"kubernetes.io/projected/44bb2522-ddd4-4fbc-ba37-68fb67c3d072-kube-api-access-cqhw8\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:40:16.026454 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.026436    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-76745cd4f8-lhchc_44bb2522-ddd4-4fbc-ba37-68fb67c3d072/main/0.log"
May 25 13:40:16.026833 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.026809    2573 generic.go:358] "Generic (PLEG): container finished" podID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerID="f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef" exitCode=137
May 25 13:40:16.026934 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.026868    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" event={"ID":"44bb2522-ddd4-4fbc-ba37-68fb67c3d072","Type":"ContainerDied","Data":"f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef"}
May 25 13:40:16.026934 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.026878    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"
May 25 13:40:16.026934 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.026905    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc" event={"ID":"44bb2522-ddd4-4fbc-ba37-68fb67c3d072","Type":"ContainerDied","Data":"d8817edfefa1107cd98ec3aa04556037fe5beccdac6e41a11ce54076de9f3bfe"}
May 25 13:40:16.026934 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.026925    2573 scope.go:117] "RemoveContainer" containerID="f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef"
May 25 13:40:16.035981 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.035964    2573 scope.go:117] "RemoveContainer" containerID="e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68"
May 25 13:40:16.047461 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.047445    2573 scope.go:117] "RemoveContainer" containerID="f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef"
May 25 13:40:16.047757 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:40:16.047734    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef\": container with ID starting with f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef not found: ID does not exist" containerID="f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef"
May 25 13:40:16.047830 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.047767    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef"} err="failed to get container status \"f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef\": rpc error: code = NotFound desc = could not find container \"f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef\": container with ID starting with f973af8c6c4f45194f46c49d1ea825ad612275736a726bbaf7449a5c168b12ef not found: ID does not exist"
May 25 13:40:16.047830 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.047791    2573 scope.go:117] "RemoveContainer" containerID="e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68"
May 25 13:40:16.048040 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:40:16.048021    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68\": container with ID starting with e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68 not found: ID does not exist" containerID="e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68"
May 25 13:40:16.048092 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.048047    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68"} err="failed to get container status \"e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68\": rpc error: code = NotFound desc = could not find container \"e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68\": container with ID starting with e40c94ef73cfb68c6c9b30ab74dcd294451cd809db92a33f7b5cdf7642ff0b68 not found: ID does not exist"
May 25 13:40:16.049375 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.049353    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"]
May 25 13:40:16.054992 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.054922    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8001/health\": dial tcp 10.132.0.33:8001: connect: connection refused"
May 25 13:40:16.055180 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.055151    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-lhchc"]
May 25 13:40:16.068264 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:16.068240    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:40:17.338483 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:17.338454    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" path="/var/lib/kubelet/pods/44bb2522-ddd4-4fbc-ba37-68fb67c3d072/volumes"
May 25 13:40:26.055717 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:26.055661    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8001/health\": dial tcp 10.132.0.33:8001: connect: connection refused"
May 25 13:40:33.386928 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:33.386899    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:40:33.389811 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:33.389790    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:40:36.055380 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:36.055331    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8001/health\": dial tcp 10.132.0.33:8001: connect: connection refused"
May 25 13:40:46.055400 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:46.055342    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8001/health\": dial tcp 10.132.0.33:8001: connect: connection refused"
May 25 13:40:56.055196 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:40:56.055144    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8001/health\": dial tcp 10.132.0.33:8001: connect: connection refused"
May 25 13:41:06.055341 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:41:06.055294    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8001/health\": dial tcp 10.132.0.33:8001: connect: connection refused"
May 25 13:41:16.055574 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:41:16.055452    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8001/health\": dial tcp 10.132.0.33:8001: connect: connection refused"
May 25 13:41:26.054898 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:41:26.054847    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8001/health\": dial tcp 10.132.0.33:8001: connect: connection refused"
May 25 13:41:36.064145 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:41:36.064116    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:41:36.076654 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:41:36.076633    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:42:01.393681 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:01.393615    2573 generic.go:358] "Generic (PLEG): container finished" podID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerID="d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020" exitCode=0
May 25 13:42:01.393681 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:01.393663    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" event={"ID":"686f5ed7-e758-489f-aa12-0eeb4c756636","Type":"ContainerDied","Data":"d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020"}
May 25 13:42:01.394775 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:01.394758    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:42:02.399024 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:02.398992    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" event={"ID":"686f5ed7-e758-489f-aa12-0eeb4c756636","Type":"ContainerStarted","Data":"d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd"}
May 25 13:42:02.420492 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:02.420441    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podStartSLOduration=127.420423058 podStartE2EDuration="2m7.420423058s" podCreationTimestamp="2026-05-25 13:39:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:42:02.418329515 +0000 UTC m=+2189.679932345" watchObservedRunningTime="2026-05-25 13:42:02.420423058 +0000 UTC m=+2189.682025890"
May 25 13:42:06.061617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:06.061581    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:42:06.062057 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:06.061628    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:42:06.062929 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:06.062899    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
May 25 13:42:16.061829 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:16.061788    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
May 25 13:42:26.062008 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:26.061964    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
May 25 13:42:36.061987 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:36.061940    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
May 25 13:42:46.062135 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:46.062042    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
May 25 13:42:56.061825 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:42:56.061781    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
May 25 13:43:06.061836 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:06.061799    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
May 25 13:43:16.062158 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:16.062113    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
May 25 13:43:26.062089 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:26.062039    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
May 25 13:43:36.073672 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:36.073644    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:43:36.082408 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:36.082382    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:43:44.130307 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:44.130271    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"]
May 25 13:43:44.130804 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:44.130659    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main" containerID="cri-o://8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061" gracePeriod=30
May 25 13:43:44.133284 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:44.133258    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"]
May 25 13:43:44.133538 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:44.133514    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main" containerID="cri-o://d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd" gracePeriod=30
May 25 13:43:48.573672 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.573625    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"]
May 25 13:43:48.574193 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.574127    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="storage-initializer"
May 25 13:43:48.574193 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.574147    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="storage-initializer"
May 25 13:43:48.574193 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.574165    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main"
May 25 13:43:48.574193 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.574174    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main"
May 25 13:43:48.574393 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.574274    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="44bb2522-ddd4-4fbc-ba37-68fb67c3d072" containerName="main"
May 25 13:43:48.578007 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.577983    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"]
May 25 13:43:48.578142 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.578129    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.580901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.580868    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 25 13:43:48.581554 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.581522    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.581677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.581656    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-x2szq\""
May 25 13:43:48.598737 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.598716    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"]
May 25 13:43:48.600051 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.600032    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"]
May 25 13:43:48.626496 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626471    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.626613 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626500    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpxw7\" (UniqueName: \"kubernetes.io/projected/a054017e-3e80-42f6-b448-315eb7107638-kube-api-access-bpxw7\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.626613 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626524    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.626613 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpp5\" (UniqueName: \"kubernetes.io/projected/df534de3-b090-4761-a621-c52543056dc0-kube-api-access-4gpp5\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.626726 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626613    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.626726 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626659    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.626726 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626712    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a054017e-3e80-42f6-b448-315eb7107638-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.626831 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626746    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.626831 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626761    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-dshm\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.626831 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626777    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.626942 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626828    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-home\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.626942 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626857    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df534de3-b090-4761-a621-c52543056dc0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.626942 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.626942 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.626899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.727651 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.727632    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.727765 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.727660    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bpxw7\" (UniqueName: \"kubernetes.io/projected/a054017e-3e80-42f6-b448-315eb7107638-kube-api-access-bpxw7\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.727765 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.727683    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.727874 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.727789    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpp5\" (UniqueName: \"kubernetes.io/projected/df534de3-b090-4761-a621-c52543056dc0-kube-api-access-4gpp5\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.727874 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.727811    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.727874 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.727827    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.727874 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.727866    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a054017e-3e80-42f6-b448-315eb7107638-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.728079 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.727919    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.728079 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.727946    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-dshm\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.728079 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728069    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.728223 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728104    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-home\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.728223 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728136    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df534de3-b090-4761-a621-c52543056dc0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.728223 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728164    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.728223 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728193    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.728491 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728228    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.728491 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728269    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.728639 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728568    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.728639 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728570    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-home\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.728639 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728603    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.728795 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728736    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.728904 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728873    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.729001 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.728907    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.730041 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.730015    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.730485 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.730464    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a054017e-3e80-42f6-b448-315eb7107638-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.730603 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.730582    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-dshm\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.730841 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.730826    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df534de3-b090-4761-a621-c52543056dc0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.735886 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.735865    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpxw7\" (UniqueName: \"kubernetes.io/projected/a054017e-3e80-42f6-b448-315eb7107638-kube-api-access-bpxw7\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.736021 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.736005    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpp5\" (UniqueName: \"kubernetes.io/projected/df534de3-b090-4761-a621-c52543056dc0-kube-api-access-4gpp5\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:48.889127 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.889062    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:48.894675 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:48.894656    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:49.028450 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:49.028409    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"]
May 25 13:43:49.030479 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:43:49.030457    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda054017e_3e80_42f6_b448_315eb7107638.slice/crio-0afa0717f45efe35bb1f7bcd14e0b5076f69e8515b103f54a80226a6b477e0ae WatchSource:0}: Error finding container 0afa0717f45efe35bb1f7bcd14e0b5076f69e8515b103f54a80226a6b477e0ae: Status 404 returned error can't find the container with id 0afa0717f45efe35bb1f7bcd14e0b5076f69e8515b103f54a80226a6b477e0ae
May 25 13:43:49.045603 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:49.045580    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"]
May 25 13:43:49.047831 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:43:49.047809    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf534de3_b090_4761_a621_c52543056dc0.slice/crio-2cc785ad2a7cb8f9e5c2c75d9264dea41b570a8fabecf1ccb5d32a9e26366069 WatchSource:0}: Error finding container 2cc785ad2a7cb8f9e5c2c75d9264dea41b570a8fabecf1ccb5d32a9e26366069: Status 404 returned error can't find the container with id 2cc785ad2a7cb8f9e5c2c75d9264dea41b570a8fabecf1ccb5d32a9e26366069
May 25 13:43:49.753098 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:49.753063    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" event={"ID":"a054017e-3e80-42f6-b448-315eb7107638","Type":"ContainerStarted","Data":"3744a7f306fbc719d21e94ddbd120f2823ac066bd66b21292e9374c6b6ddc8f8"}
May 25 13:43:49.753098 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:49.753100    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" event={"ID":"a054017e-3e80-42f6-b448-315eb7107638","Type":"ContainerStarted","Data":"0afa0717f45efe35bb1f7bcd14e0b5076f69e8515b103f54a80226a6b477e0ae"}
May 25 13:43:49.753538 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:49.753135    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:43:49.754419 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:49.754392    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" event={"ID":"df534de3-b090-4761-a621-c52543056dc0","Type":"ContainerStarted","Data":"613bba6d633313b9346c5affd2b8c39be353ef2cc974037e5c6477144b15139c"}
May 25 13:43:49.754520 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:49.754423    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" event={"ID":"df534de3-b090-4761-a621-c52543056dc0","Type":"ContainerStarted","Data":"2cc785ad2a7cb8f9e5c2c75d9264dea41b570a8fabecf1ccb5d32a9e26366069"}
May 25 13:43:50.760717 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:50.760674    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" event={"ID":"a054017e-3e80-42f6-b448-315eb7107638","Type":"ContainerStarted","Data":"c040fa1b8475018960685e8aa72c0eae1632909cf827369bb3fcf61930a5f342"}
May 25 13:43:53.772016 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:53.771981    2573 generic.go:358] "Generic (PLEG): container finished" podID="df534de3-b090-4761-a621-c52543056dc0" containerID="613bba6d633313b9346c5affd2b8c39be353ef2cc974037e5c6477144b15139c" exitCode=0
May 25 13:43:53.772380 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:53.772055    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" event={"ID":"df534de3-b090-4761-a621-c52543056dc0","Type":"ContainerDied","Data":"613bba6d633313b9346c5affd2b8c39be353ef2cc974037e5c6477144b15139c"}
May 25 13:43:54.776974 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:54.776941    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" event={"ID":"df534de3-b090-4761-a621-c52543056dc0","Type":"ContainerStarted","Data":"d3469d915c434d6cd7260fbaacee0e98c71e365a21a0ffcb84b45b2aecf44425"}
May 25 13:43:54.799063 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:54.799011    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podStartSLOduration=6.798994665 podStartE2EDuration="6.798994665s" podCreationTimestamp="2026-05-25 13:43:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:43:54.796820332 +0000 UTC m=+2302.058423163" watchObservedRunningTime="2026-05-25 13:43:54.798994665 +0000 UTC m=+2302.060597497"
May 25 13:43:58.895330 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:58.895297    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:58.895330 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:58.895333    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:43:58.897076 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:43:58.897048    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
May 25 13:44:01.778786 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:01.778757    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:44:08.895323 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:08.895274    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
May 25 13:44:14.131349 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.131304    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="llm-d-routing-sidecar" containerID="cri-o://2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5" gracePeriod=2
May 25 13:44:14.433071 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.433048    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg_b486f451-c7a6-4d65-906e-1ffe3ce69b14/main/0.log"
May 25 13:44:14.433773 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.433755    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:44:14.437081 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.437060    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:44:14.553244 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553215    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-model-cache\") pod \"686f5ed7-e758-489f-aa12-0eeb4c756636\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") "
May 25 13:44:14.553424 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553267    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jttbs\" (UniqueName: \"kubernetes.io/projected/686f5ed7-e758-489f-aa12-0eeb4c756636-kube-api-access-jttbs\") pod \"686f5ed7-e758-489f-aa12-0eeb4c756636\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") "
May 25 13:44:14.553424 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553300    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-tmp-dir\") pod \"686f5ed7-e758-489f-aa12-0eeb4c756636\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") "
May 25 13:44:14.553424 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553317    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-home\") pod \"686f5ed7-e758-489f-aa12-0eeb4c756636\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") "
May 25 13:44:14.553424 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553336    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5lfs\" (UniqueName: \"kubernetes.io/projected/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kube-api-access-x5lfs\") pod \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") "
May 25 13:44:14.553424 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553361    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-model-cache\") pod \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") "
May 25 13:44:14.553424 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553383    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tmp-dir\") pod \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") "
May 25 13:44:14.553424 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553421    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-kserve-provision-location\") pod \"686f5ed7-e758-489f-aa12-0eeb4c756636\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") "
May 25 13:44:14.553886 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553457    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-home\") pod \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") "
May 25 13:44:14.553886 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553482    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686f5ed7-e758-489f-aa12-0eeb4c756636-tls-certs\") pod \"686f5ed7-e758-489f-aa12-0eeb4c756636\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") "
May 25 13:44:14.553886 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553517    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-dshm\") pod \"686f5ed7-e758-489f-aa12-0eeb4c756636\" (UID: \"686f5ed7-e758-489f-aa12-0eeb4c756636\") "
May 25 13:44:14.553886 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553542    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kserve-provision-location\") pod \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") "
May 25 13:44:14.553886 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553596    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-dshm\") pod \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") "
May 25 13:44:14.553886 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.553628    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tls-certs\") pod \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\" (UID: \"b486f451-c7a6-4d65-906e-1ffe3ce69b14\") "
May 25 13:44:14.554163 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.554017    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-home" (OuterVolumeSpecName: "home") pod "686f5ed7-e758-489f-aa12-0eeb4c756636" (UID: "686f5ed7-e758-489f-aa12-0eeb4c756636"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:14.554214 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.554198    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-model-cache" (OuterVolumeSpecName: "model-cache") pod "b486f451-c7a6-4d65-906e-1ffe3ce69b14" (UID: "b486f451-c7a6-4d65-906e-1ffe3ce69b14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:14.555725 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.555696    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-model-cache" (OuterVolumeSpecName: "model-cache") pod "686f5ed7-e758-489f-aa12-0eeb4c756636" (UID: "686f5ed7-e758-489f-aa12-0eeb4c756636"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:14.556303 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.556280    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-home" (OuterVolumeSpecName: "home") pod "b486f451-c7a6-4d65-906e-1ffe3ce69b14" (UID: "b486f451-c7a6-4d65-906e-1ffe3ce69b14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:14.556802 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.556759    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kube-api-access-x5lfs" (OuterVolumeSpecName: "kube-api-access-x5lfs") pod "b486f451-c7a6-4d65-906e-1ffe3ce69b14" (UID: "b486f451-c7a6-4d65-906e-1ffe3ce69b14"). InnerVolumeSpecName "kube-api-access-x5lfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:44:14.557024 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.556985    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686f5ed7-e758-489f-aa12-0eeb4c756636-kube-api-access-jttbs" (OuterVolumeSpecName: "kube-api-access-jttbs") pod "686f5ed7-e758-489f-aa12-0eeb4c756636" (UID: "686f5ed7-e758-489f-aa12-0eeb4c756636"). InnerVolumeSpecName "kube-api-access-jttbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:44:14.557141 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.557105    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-dshm" (OuterVolumeSpecName: "dshm") pod "686f5ed7-e758-489f-aa12-0eeb4c756636" (UID: "686f5ed7-e758-489f-aa12-0eeb4c756636"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:14.559144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.559113    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b486f451-c7a6-4d65-906e-1ffe3ce69b14" (UID: "b486f451-c7a6-4d65-906e-1ffe3ce69b14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:44:14.559345 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.559319    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-dshm" (OuterVolumeSpecName: "dshm") pod "b486f451-c7a6-4d65-906e-1ffe3ce69b14" (UID: "b486f451-c7a6-4d65-906e-1ffe3ce69b14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:14.559739 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.559719    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686f5ed7-e758-489f-aa12-0eeb4c756636-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "686f5ed7-e758-489f-aa12-0eeb4c756636" (UID: "686f5ed7-e758-489f-aa12-0eeb4c756636"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:44:14.569182 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.569148    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b486f451-c7a6-4d65-906e-1ffe3ce69b14" (UID: "b486f451-c7a6-4d65-906e-1ffe3ce69b14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:14.573874 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.573850    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "686f5ed7-e758-489f-aa12-0eeb4c756636" (UID: "686f5ed7-e758-489f-aa12-0eeb4c756636"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:14.617468 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.617437    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "686f5ed7-e758-489f-aa12-0eeb4c756636" (UID: "686f5ed7-e758-489f-aa12-0eeb4c756636"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:14.622244 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.622219    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b486f451-c7a6-4d65-906e-1ffe3ce69b14" (UID: "b486f451-c7a6-4d65-906e-1ffe3ce69b14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:14.654457 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654404    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jttbs\" (UniqueName: \"kubernetes.io/projected/686f5ed7-e758-489f-aa12-0eeb4c756636-kube-api-access-jttbs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654457 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654425    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654457 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654434    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654457 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654442    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5lfs\" (UniqueName: \"kubernetes.io/projected/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kube-api-access-x5lfs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654457 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654453    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654699 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654463    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654699 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654473    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654699 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654483    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654699 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654492    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686f5ed7-e758-489f-aa12-0eeb4c756636-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654699 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654500    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654699 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654508    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654699 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654515    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b486f451-c7a6-4d65-906e-1ffe3ce69b14-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654699 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654523    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b486f451-c7a6-4d65-906e-1ffe3ce69b14-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.654699 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.654530    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686f5ed7-e758-489f-aa12-0eeb4c756636-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:44:14.848147 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.848113    2573 generic.go:358] "Generic (PLEG): container finished" podID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerID="d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd" exitCode=137
May 25 13:44:14.848348 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.848160    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" event={"ID":"686f5ed7-e758-489f-aa12-0eeb4c756636","Type":"ContainerDied","Data":"d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd"}
May 25 13:44:14.848348 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.848190    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"
May 25 13:44:14.848348 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.848198    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7" event={"ID":"686f5ed7-e758-489f-aa12-0eeb4c756636","Type":"ContainerDied","Data":"51ed1f57cad4a9df59c1b560c9a40e2212cd5f011d5317ce5a350b55b09b0fbf"}
May 25 13:44:14.848348 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.848214    2573 scope.go:117] "RemoveContainer" containerID="d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd"
May 25 13:44:14.849757 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.849741    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg_b486f451-c7a6-4d65-906e-1ffe3ce69b14/main/0.log"
May 25 13:44:14.850452 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.850429    2573 generic.go:358] "Generic (PLEG): container finished" podID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerID="8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061" exitCode=137
May 25 13:44:14.850452 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.850450    2573 generic.go:358] "Generic (PLEG): container finished" podID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerID="2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5" exitCode=0
May 25 13:44:14.850677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.850478    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" event={"ID":"b486f451-c7a6-4d65-906e-1ffe3ce69b14","Type":"ContainerDied","Data":"8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061"}
May 25 13:44:14.850677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.850497    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"
May 25 13:44:14.850677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.850504    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" event={"ID":"b486f451-c7a6-4d65-906e-1ffe3ce69b14","Type":"ContainerDied","Data":"2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5"}
May 25 13:44:14.850677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.850519    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg" event={"ID":"b486f451-c7a6-4d65-906e-1ffe3ce69b14","Type":"ContainerDied","Data":"c30ca3bdacc1746a3f5c76b205aed477249b4c40d62e89f7944d80ae5cb4e913"}
May 25 13:44:14.857469 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.857337    2573 scope.go:117] "RemoveContainer" containerID="d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020"
May 25 13:44:14.872214 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.872192    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"]
May 25 13:44:14.876483 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.876463    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-5585kq7"]
May 25 13:44:14.878563 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.878534    2573 scope.go:117] "RemoveContainer" containerID="d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd"
May 25 13:44:14.878836 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:44:14.878815    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd\": container with ID starting with d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd not found: ID does not exist" containerID="d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd"
May 25 13:44:14.878905 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.878842    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd"} err="failed to get container status \"d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd\": rpc error: code = NotFound desc = could not find container \"d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd\": container with ID starting with d38606117fb581d22f0ed6a4a74c8b377790152df0115efd7d6c01abbbcfc1bd not found: ID does not exist"
May 25 13:44:14.878905 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.878858    2573 scope.go:117] "RemoveContainer" containerID="d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020"
May 25 13:44:14.879077 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:44:14.879061    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020\": container with ID starting with d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020 not found: ID does not exist" containerID="d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020"
May 25 13:44:14.879127 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.879081    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020"} err="failed to get container status \"d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020\": rpc error: code = NotFound desc = could not find container \"d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020\": container with ID starting with d52f4db2733339fae7d4f71de82460c0bcabd00677c24ae9fc43309d935d0020 not found: ID does not exist"
May 25 13:44:14.879127 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.879095    2573 scope.go:117] "RemoveContainer" containerID="8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061"
May 25 13:44:14.887641 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.887623    2573 scope.go:117] "RemoveContainer" containerID="cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150"
May 25 13:44:14.888683 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.888663    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"]
May 25 13:44:14.892163 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.892128    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf26tfg"]
May 25 13:44:14.897102 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.897075    2573 scope.go:117] "RemoveContainer" containerID="2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5"
May 25 13:44:14.903981 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.903964    2573 scope.go:117] "RemoveContainer" containerID="8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061"
May 25 13:44:14.904237 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:44:14.904220    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061\": container with ID starting with 8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061 not found: ID does not exist" containerID="8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061"
May 25 13:44:14.904309 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.904247    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061"} err="failed to get container status \"8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061\": rpc error: code = NotFound desc = could not find container \"8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061\": container with ID starting with 8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061 not found: ID does not exist"
May 25 13:44:14.904309 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.904271    2573 scope.go:117] "RemoveContainer" containerID="cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150"
May 25 13:44:14.904534 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:44:14.904493    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150\": container with ID starting with cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150 not found: ID does not exist" containerID="cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150"
May 25 13:44:14.904534 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.904518    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150"} err="failed to get container status \"cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150\": rpc error: code = NotFound desc = could not find container \"cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150\": container with ID starting with cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150 not found: ID does not exist"
May 25 13:44:14.904648 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.904534    2573 scope.go:117] "RemoveContainer" containerID="2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5"
May 25 13:44:14.904819 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:44:14.904800    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5\": container with ID starting with 2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5 not found: ID does not exist" containerID="2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5"
May 25 13:44:14.904859 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.904825    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5"} err="failed to get container status \"2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5\": rpc error: code = NotFound desc = could not find container \"2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5\": container with ID starting with 2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5 not found: ID does not exist"
May 25 13:44:14.904859 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.904845    2573 scope.go:117] "RemoveContainer" containerID="8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061"
May 25 13:44:14.905085 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.905069    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061"} err="failed to get container status \"8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061\": rpc error: code = NotFound desc = could not find container \"8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061\": container with ID starting with 8f4b42f8cb57365bec88edc91046c97132ed721958b27cbfe04dc81b8eb7d061 not found: ID does not exist"
May 25 13:44:14.905138 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.905085    2573 scope.go:117] "RemoveContainer" containerID="cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150"
May 25 13:44:14.905277 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.905261    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150"} err="failed to get container status \"cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150\": rpc error: code = NotFound desc = could not find container \"cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150\": container with ID starting with cd03ab11efec1359ff0afb8f3a27b48fce32058fc4a1202ed3ac13137da60150 not found: ID does not exist"
May 25 13:44:14.905333 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.905283    2573 scope.go:117] "RemoveContainer" containerID="2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5"
May 25 13:44:14.905475 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:14.905460    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5"} err="failed to get container status \"2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5\": rpc error: code = NotFound desc = could not find container \"2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5\": container with ID starting with 2b27115c705cdb3fa42d91fddfb314aa1dc229020e3644b138ff7c079384e7f5 not found: ID does not exist"
May 25 13:44:15.343565 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:15.343516    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" path="/var/lib/kubelet/pods/686f5ed7-e758-489f-aa12-0eeb4c756636/volumes"
May 25 13:44:15.344163 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:15.344142    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" path="/var/lib/kubelet/pods/b486f451-c7a6-4d65-906e-1ffe3ce69b14/volumes"
May 25 13:44:18.895589 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:18.895527    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
May 25 13:44:28.895722 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:28.895671    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
May 25 13:44:38.895166 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:38.895119    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
May 25 13:44:48.895618 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:48.895566    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
May 25 13:44:58.895478 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:44:58.895427    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
May 25 13:45:08.895001 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:08.894953    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
May 25 13:45:18.895973 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:18.895935    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
May 25 13:45:28.907870 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:28.907840    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:45:28.915620 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:28.915596    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:45:33.414041 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:33.414017    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:45:33.419005 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:33.418986    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:45:43.467029 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:43.466997    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"]
May 25 13:45:43.467580 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:43.467357    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c" podUID="1c2df3ec-9f0f-4734-acad-c0a9b086bed7" containerName="storage-initializer" containerID="cri-o://51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a" gracePeriod=30
May 25 13:45:56.113290 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113250    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"]
May 25 13:45:56.113787 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113726    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main"
May 25 13:45:56.113787 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113745    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main"
May 25 13:45:56.113787 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113757    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main"
May 25 13:45:56.113787 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113765    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main"
May 25 13:45:56.113787 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113777    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="llm-d-routing-sidecar"
May 25 13:45:56.113787 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113785    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="llm-d-routing-sidecar"
May 25 13:45:56.114108 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113797    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="storage-initializer"
May 25 13:45:56.114108 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113805    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="storage-initializer"
May 25 13:45:56.114108 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113831    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="storage-initializer"
May 25 13:45:56.114108 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113840    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="storage-initializer"
May 25 13:45:56.114108 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113922    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="llm-d-routing-sidecar"
May 25 13:45:56.114108 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113941    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="b486f451-c7a6-4d65-906e-1ffe3ce69b14" containerName="main"
May 25 13:45:56.114108 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.113951    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="686f5ed7-e758-489f-aa12-0eeb4c756636" containerName="main"
May 25 13:45:56.117462 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.117439    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.121083 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.121064    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"conv-test-round-trip-kserve-self-signed-certs\""
May 25 13:45:56.128809 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.128740    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"]
May 25 13:45:56.248325 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.248300    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbnxj\" (UniqueName: \"kubernetes.io/projected/b625c473-5747-45d8-ae29-81cec302e274-kube-api-access-zbnxj\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.248422 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.248339    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-home\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.248422 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.248356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-tmp-dir\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.248422 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.248397    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b625c473-5747-45d8-ae29-81cec302e274-tls-certs\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.248422 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.248412    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-kserve-provision-location\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.248577 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.248495    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-model-cache\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.248577 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.248511    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-dshm\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.349365 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349342    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-model-cache\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.349449 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349371    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-dshm\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.349449 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349418    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnxj\" (UniqueName: \"kubernetes.io/projected/b625c473-5747-45d8-ae29-81cec302e274-kube-api-access-zbnxj\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.349520 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349455    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-home\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.349520 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349480    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-tmp-dir\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.349520 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349508    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b625c473-5747-45d8-ae29-81cec302e274-tls-certs\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.349691 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349653    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-kserve-provision-location\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.349795 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349774    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-model-cache\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.349926 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349904    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-tmp-dir\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.349926 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349920    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-home\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.350031 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.349965    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-kserve-provision-location\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.351724 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.351694    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-dshm\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.351902 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.351884    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b625c473-5747-45d8-ae29-81cec302e274-tls-certs\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.357267 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.357245    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnxj\" (UniqueName: \"kubernetes.io/projected/b625c473-5747-45d8-ae29-81cec302e274-kube-api-access-zbnxj\") pod \"conv-test-round-trip-kserve-6bf87c4986-5hmv5\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.429074 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.429023    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:45:56.552748 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:56.552717    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"]
May 25 13:45:56.555224 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:45:56.555194    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb625c473_5747_45d8_ae29_81cec302e274.slice/crio-4a5f3f7b2dce2d079e26b6953db994d05d1ee27dbe894ba623b7f8cbf2d5ebc6 WatchSource:0}: Error finding container 4a5f3f7b2dce2d079e26b6953db994d05d1ee27dbe894ba623b7f8cbf2d5ebc6: Status 404 returned error can't find the container with id 4a5f3f7b2dce2d079e26b6953db994d05d1ee27dbe894ba623b7f8cbf2d5ebc6
May 25 13:45:57.194411 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:57.194373    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5" event={"ID":"b625c473-5747-45d8-ae29-81cec302e274","Type":"ContainerStarted","Data":"6748e3f76bcf3b80907ef907e1e4760805fdca22034f0aa5ddf1bc81696281cb"}
May 25 13:45:57.194834 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:45:57.194415    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5" event={"ID":"b625c473-5747-45d8-ae29-81cec302e274","Type":"ContainerStarted","Data":"4a5f3f7b2dce2d079e26b6953db994d05d1ee27dbe894ba623b7f8cbf2d5ebc6"}
May 25 13:46:03.914211 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:03.914169    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"]
May 25 13:46:03.920814 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:03.920796    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:03.923202 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:03.923186    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 25 13:46:03.929805 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:03.929780    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"]
May 25 13:46:04.012346 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.012325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc009044-fa41-44f2-9280-f0074f407cd7-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.012466 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.012362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-home\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.012528 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.012463    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.012528 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.012503    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.012642 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.012565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.012642 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.012594    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.012642 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.012612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfnzp\" (UniqueName: \"kubernetes.io/projected/cc009044-fa41-44f2-9280-f0074f407cd7-kube-api-access-bfnzp\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.113684 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.113654    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc009044-fa41-44f2-9280-f0074f407cd7-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.113901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.113700    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-home\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.113901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.113749    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.113901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.113778    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.113901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.113810    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.113901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.113860    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.113901 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.113881    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bfnzp\" (UniqueName: \"kubernetes.io/projected/cc009044-fa41-44f2-9280-f0074f407cd7-kube-api-access-bfnzp\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.114215 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.114190    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.114275 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.114231    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.114330 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.114273    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.114330 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.114314    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-home\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.116055 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.116033    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.116271 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.116252    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc009044-fa41-44f2-9280-f0074f407cd7-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.121651 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.121625    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfnzp\" (UniqueName: \"kubernetes.io/projected/cc009044-fa41-44f2-9280-f0074f407cd7-kube-api-access-bfnzp\") pod \"stop-feature-test-kserve-5474d9f6f4-lxtw9\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.232124 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.232052    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:04.357444 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:04.357419    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"]
May 25 13:46:04.359254 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:46:04.359227    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc009044_fa41_44f2_9280_f0074f407cd7.slice/crio-417cbe494cbe5b54a78709617db350483651aa0b4dd42a8ca591359ce10c7fd5 WatchSource:0}: Error finding container 417cbe494cbe5b54a78709617db350483651aa0b4dd42a8ca591359ce10c7fd5: Status 404 returned error can't find the container with id 417cbe494cbe5b54a78709617db350483651aa0b4dd42a8ca591359ce10c7fd5
May 25 13:46:05.223350 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:05.223314    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" event={"ID":"cc009044-fa41-44f2-9280-f0074f407cd7","Type":"ContainerStarted","Data":"386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668"}
May 25 13:46:05.223350 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:05.223353    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" event={"ID":"cc009044-fa41-44f2-9280-f0074f407cd7","Type":"ContainerStarted","Data":"417cbe494cbe5b54a78709617db350483651aa0b4dd42a8ca591359ce10c7fd5"}
May 25 13:46:07.709135 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:07.709097    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"]
May 25 13:46:07.709489 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:07.709347    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5" podUID="b625c473-5747-45d8-ae29-81cec302e274" containerName="storage-initializer" containerID="cri-o://6748e3f76bcf3b80907ef907e1e4760805fdca22034f0aa5ddf1bc81696281cb" gracePeriod=30
May 25 13:46:13.252714 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.252676    2573 generic.go:358] "Generic (PLEG): container finished" podID="cc009044-fa41-44f2-9280-f0074f407cd7" containerID="386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668" exitCode=0
May 25 13:46:13.253023 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.252741    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" event={"ID":"cc009044-fa41-44f2-9280-f0074f407cd7","Type":"ContainerDied","Data":"386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668"}
May 25 13:46:13.658599 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.658575    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-84f494576-gk56c_1c2df3ec-9f0f-4734-acad-c0a9b086bed7/storage-initializer/0.log"
May 25 13:46:13.658704 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.658658    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:46:13.799677 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.799646    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kserve-provision-location\") pod \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") "
May 25 13:46:13.799848 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.799723    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tmp-dir\") pod \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") "
May 25 13:46:13.799848 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.799760    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tls-certs\") pod \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") "
May 25 13:46:13.799848 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.799777    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-home\") pod \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") "
May 25 13:46:13.799848 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.799794    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-model-cache\") pod \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") "
May 25 13:46:13.799848 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.799831    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-dshm\") pod \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") "
May 25 13:46:13.800116 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.799860    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf2gq\" (UniqueName: \"kubernetes.io/projected/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kube-api-access-wf2gq\") pod \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\" (UID: \"1c2df3ec-9f0f-4734-acad-c0a9b086bed7\") "
May 25 13:46:13.800116 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.800038    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c2df3ec-9f0f-4734-acad-c0a9b086bed7" (UID: "1c2df3ec-9f0f-4734-acad-c0a9b086bed7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:13.800116 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.800047    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-home" (OuterVolumeSpecName: "home") pod "1c2df3ec-9f0f-4734-acad-c0a9b086bed7" (UID: "1c2df3ec-9f0f-4734-acad-c0a9b086bed7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:13.800116 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.800062    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c2df3ec-9f0f-4734-acad-c0a9b086bed7" (UID: "1c2df3ec-9f0f-4734-acad-c0a9b086bed7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:13.800318 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.800119    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:13.800318 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.800138    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:13.800318 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.800149    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:13.802260 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.802216    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c2df3ec-9f0f-4734-acad-c0a9b086bed7" (UID: "1c2df3ec-9f0f-4734-acad-c0a9b086bed7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:46:13.802374 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.802295    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kube-api-access-wf2gq" (OuterVolumeSpecName: "kube-api-access-wf2gq") pod "1c2df3ec-9f0f-4734-acad-c0a9b086bed7" (UID: "1c2df3ec-9f0f-4734-acad-c0a9b086bed7"). InnerVolumeSpecName "kube-api-access-wf2gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:46:13.802374 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.802308    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-dshm" (OuterVolumeSpecName: "dshm") pod "1c2df3ec-9f0f-4734-acad-c0a9b086bed7" (UID: "1c2df3ec-9f0f-4734-acad-c0a9b086bed7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:13.842370 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.842340    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c2df3ec-9f0f-4734-acad-c0a9b086bed7" (UID: "1c2df3ec-9f0f-4734-acad-c0a9b086bed7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:13.901534 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.901510    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:13.901534 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.901534    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf2gq\" (UniqueName: \"kubernetes.io/projected/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kube-api-access-wf2gq\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:13.901714 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.901557    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:13.901714 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:13.901566    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2df3ec-9f0f-4734-acad-c0a9b086bed7-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:14.257388 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.257366    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-84f494576-gk56c_1c2df3ec-9f0f-4734-acad-c0a9b086bed7/storage-initializer/0.log"
May 25 13:46:14.257807 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.257404    2573 generic.go:358] "Generic (PLEG): container finished" podID="1c2df3ec-9f0f-4734-acad-c0a9b086bed7" containerID="51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a" exitCode=137
May 25 13:46:14.257807 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.257477    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"
May 25 13:46:14.257807 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.257472    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c" event={"ID":"1c2df3ec-9f0f-4734-acad-c0a9b086bed7","Type":"ContainerDied","Data":"51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a"}
May 25 13:46:14.257807 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.257604    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c" event={"ID":"1c2df3ec-9f0f-4734-acad-c0a9b086bed7","Type":"ContainerDied","Data":"9d89b9e8a7041f771825fcc6f36437bfece21960dee92fdc3b9f3b5322d10cd0"}
May 25 13:46:14.257807 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.257630    2573 scope.go:117] "RemoveContainer" containerID="51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a"
May 25 13:46:14.259396 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.259372    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" event={"ID":"cc009044-fa41-44f2-9280-f0074f407cd7","Type":"ContainerStarted","Data":"e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13"}
May 25 13:46:14.280194 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.280136    2573 scope.go:117] "RemoveContainer" containerID="51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a"
May 25 13:46:14.280810 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:46:14.280782    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a\": container with ID starting with 51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a not found: ID does not exist" containerID="51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a"
May 25 13:46:14.280915 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.280822    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a"} err="failed to get container status \"51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a\": rpc error: code = NotFound desc = could not find container \"51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a\": container with ID starting with 51904c44f425eb498c25616c2068f3e74eb22bef9d849313b163093027aa503a not found: ID does not exist"
May 25 13:46:14.282954 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.282913    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" podStartSLOduration=11.282901233 podStartE2EDuration="11.282901233s" podCreationTimestamp="2026-05-25 13:46:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:46:14.280003731 +0000 UTC m=+2441.541606563" watchObservedRunningTime="2026-05-25 13:46:14.282901233 +0000 UTC m=+2441.544504057"
May 25 13:46:14.307513 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.307489    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"]
May 25 13:46:14.314595 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:14.314570    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-gk56c"]
May 25 13:46:15.338259 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:15.338230    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2df3ec-9f0f-4734-acad-c0a9b086bed7" path="/var/lib/kubelet/pods/1c2df3ec-9f0f-4734-acad-c0a9b086bed7/volumes"
May 25 13:46:24.232960 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:24.232921    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:24.234336 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:24.234301    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:46:24.234572 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:24.234522    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 25 13:46:34.232930 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:34.232884    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 25 13:46:38.343973 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.343944    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-round-trip-kserve-6bf87c4986-5hmv5_b625c473-5747-45d8-ae29-81cec302e274/storage-initializer/0.log"
May 25 13:46:38.344285 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.343998    2573 generic.go:358] "Generic (PLEG): container finished" podID="b625c473-5747-45d8-ae29-81cec302e274" containerID="6748e3f76bcf3b80907ef907e1e4760805fdca22034f0aa5ddf1bc81696281cb" exitCode=137
May 25 13:46:38.344285 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.344071    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5" event={"ID":"b625c473-5747-45d8-ae29-81cec302e274","Type":"ContainerDied","Data":"6748e3f76bcf3b80907ef907e1e4760805fdca22034f0aa5ddf1bc81696281cb"}
May 25 13:46:38.358679 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.358658    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-round-trip-kserve-6bf87c4986-5hmv5_b625c473-5747-45d8-ae29-81cec302e274/storage-initializer/0.log"
May 25 13:46:38.358788 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.358721    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:46:38.391617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.391589    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-model-cache\") pod \"b625c473-5747-45d8-ae29-81cec302e274\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") "
May 25 13:46:38.391617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.391620    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-dshm\") pod \"b625c473-5747-45d8-ae29-81cec302e274\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") "
May 25 13:46:38.391795 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.391677    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-home\") pod \"b625c473-5747-45d8-ae29-81cec302e274\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") "
May 25 13:46:38.391795 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.391705    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b625c473-5747-45d8-ae29-81cec302e274-tls-certs\") pod \"b625c473-5747-45d8-ae29-81cec302e274\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") "
May 25 13:46:38.391795 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.391759    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-tmp-dir\") pod \"b625c473-5747-45d8-ae29-81cec302e274\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") "
May 25 13:46:38.391939 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.391793    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbnxj\" (UniqueName: \"kubernetes.io/projected/b625c473-5747-45d8-ae29-81cec302e274-kube-api-access-zbnxj\") pod \"b625c473-5747-45d8-ae29-81cec302e274\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") "
May 25 13:46:38.391939 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.391842    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-kserve-provision-location\") pod \"b625c473-5747-45d8-ae29-81cec302e274\" (UID: \"b625c473-5747-45d8-ae29-81cec302e274\") "
May 25 13:46:38.391939 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.391892    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-model-cache" (OuterVolumeSpecName: "model-cache") pod "b625c473-5747-45d8-ae29-81cec302e274" (UID: "b625c473-5747-45d8-ae29-81cec302e274"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:38.392090 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.391982    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-home" (OuterVolumeSpecName: "home") pod "b625c473-5747-45d8-ae29-81cec302e274" (UID: "b625c473-5747-45d8-ae29-81cec302e274"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:38.392090 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.392065    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b625c473-5747-45d8-ae29-81cec302e274" (UID: "b625c473-5747-45d8-ae29-81cec302e274"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:38.392443 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.392181    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:38.392443 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.392215    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:38.392443 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.392230    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:38.393965 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.393942    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-dshm" (OuterVolumeSpecName: "dshm") pod "b625c473-5747-45d8-ae29-81cec302e274" (UID: "b625c473-5747-45d8-ae29-81cec302e274"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:38.394231 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.394214    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b625c473-5747-45d8-ae29-81cec302e274-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b625c473-5747-45d8-ae29-81cec302e274" (UID: "b625c473-5747-45d8-ae29-81cec302e274"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:46:38.394627 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.394598    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b625c473-5747-45d8-ae29-81cec302e274-kube-api-access-zbnxj" (OuterVolumeSpecName: "kube-api-access-zbnxj") pod "b625c473-5747-45d8-ae29-81cec302e274" (UID: "b625c473-5747-45d8-ae29-81cec302e274"). InnerVolumeSpecName "kube-api-access-zbnxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:46:38.433084 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.433038    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b625c473-5747-45d8-ae29-81cec302e274" (UID: "b625c473-5747-45d8-ae29-81cec302e274"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:38.493099 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.493077    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b625c473-5747-45d8-ae29-81cec302e274-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:38.493099 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.493100    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbnxj\" (UniqueName: \"kubernetes.io/projected/b625c473-5747-45d8-ae29-81cec302e274-kube-api-access-zbnxj\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:38.493242 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.493110    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:38.493242 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:38.493119    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b625c473-5747-45d8-ae29-81cec302e274-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:46:39.347784 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:39.347761    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-round-trip-kserve-6bf87c4986-5hmv5_b625c473-5747-45d8-ae29-81cec302e274/storage-initializer/0.log"
May 25 13:46:39.348080 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:39.347884    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"
May 25 13:46:39.348080 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:39.347885    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5" event={"ID":"b625c473-5747-45d8-ae29-81cec302e274","Type":"ContainerDied","Data":"4a5f3f7b2dce2d079e26b6953db994d05d1ee27dbe894ba623b7f8cbf2d5ebc6"}
May 25 13:46:39.348080 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:39.347993    2573 scope.go:117] "RemoveContainer" containerID="6748e3f76bcf3b80907ef907e1e4760805fdca22034f0aa5ddf1bc81696281cb"
May 25 13:46:39.381583 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:39.381555    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"]
May 25 13:46:39.385621 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:39.385598    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-6bf87c4986-5hmv5"]
May 25 13:46:41.342379 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:41.341963    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b625c473-5747-45d8-ae29-81cec302e274" path="/var/lib/kubelet/pods/b625c473-5747-45d8-ae29-81cec302e274/volumes"
May 25 13:46:44.233132 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:44.233083    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 25 13:46:54.232964 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:46:54.232915    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 25 13:47:04.233280 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:47:04.233228    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 25 13:47:14.233368 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:47:14.233325    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 25 13:47:24.232977 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:47:24.232894    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 25 13:47:34.232616 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:47:34.232577    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 25 13:47:44.241607 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:47:44.241565    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:47:44.248955 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:47:44.248934    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:47:45.294869 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:47:45.294833    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"]
May 25 13:47:45.579177 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:47:45.579068    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main" containerID="cri-o://e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13" gracePeriod=30
May 25 13:48:08.032742 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.032706    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"]
May 25 13:48:08.033180 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.033023    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2df3ec-9f0f-4734-acad-c0a9b086bed7" containerName="storage-initializer"
May 25 13:48:08.033180 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.033035    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2df3ec-9f0f-4734-acad-c0a9b086bed7" containerName="storage-initializer"
May 25 13:48:08.033180 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.033058    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b625c473-5747-45d8-ae29-81cec302e274" containerName="storage-initializer"
May 25 13:48:08.033180 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.033064    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b625c473-5747-45d8-ae29-81cec302e274" containerName="storage-initializer"
May 25 13:48:08.033180 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.033111    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c2df3ec-9f0f-4734-acad-c0a9b086bed7" containerName="storage-initializer"
May 25 13:48:08.033180 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.033123    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="b625c473-5747-45d8-ae29-81cec302e274" containerName="storage-initializer"
May 25 13:48:08.036044 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.036026    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.051241 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.051217    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"]
May 25 13:48:08.100187 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.100154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-home\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.100345 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.100212    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9ghj\" (UniqueName: \"kubernetes.io/projected/ed916505-a5f7-4764-8fca-68f8b057337d-kube-api-access-v9ghj\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.100345 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.100242    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.100345 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.100270    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed916505-a5f7-4764-8fca-68f8b057337d-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.100482 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.100363    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.100482 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.100395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.100482 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.100463    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201066 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201038    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201240 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201079    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-home\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201240 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201114    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9ghj\" (UniqueName: \"kubernetes.io/projected/ed916505-a5f7-4764-8fca-68f8b057337d-kube-api-access-v9ghj\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201240 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201141    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201240 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201170    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed916505-a5f7-4764-8fca-68f8b057337d-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201449 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201312    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201449 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201356    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201587 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201488    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201587 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201504    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-home\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201697 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201654    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.201697 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.201657    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.203486 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.203462    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.203743 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.203726    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed916505-a5f7-4764-8fca-68f8b057337d-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.208510 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.208490    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9ghj\" (UniqueName: \"kubernetes.io/projected/ed916505-a5f7-4764-8fca-68f8b057337d-kube-api-access-v9ghj\") pod \"stop-feature-test-kserve-5474d9f6f4-kj84v\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.346150 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.346063    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:08.475154 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.475121    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"]
May 25 13:48:08.478512 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:48:08.478487    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded916505_a5f7_4764_8fca_68f8b057337d.slice/crio-46026ec562809c602ded7bdc770397e141594e3094476abaa6949185251d2ba4 WatchSource:0}: Error finding container 46026ec562809c602ded7bdc770397e141594e3094476abaa6949185251d2ba4: Status 404 returned error can't find the container with id 46026ec562809c602ded7bdc770397e141594e3094476abaa6949185251d2ba4
May 25 13:48:08.480364 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.480347    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:48:08.655888 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.655816    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" event={"ID":"ed916505-a5f7-4764-8fca-68f8b057337d","Type":"ContainerStarted","Data":"4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553"}
May 25 13:48:08.655888 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:08.655858    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" event={"ID":"ed916505-a5f7-4764-8fca-68f8b057337d","Type":"ContainerStarted","Data":"46026ec562809c602ded7bdc770397e141594e3094476abaa6949185251d2ba4"}
May 25 13:48:12.671751 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:12.671723    2573 generic.go:358] "Generic (PLEG): container finished" podID="ed916505-a5f7-4764-8fca-68f8b057337d" containerID="4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553" exitCode=0
May 25 13:48:12.672195 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:12.671784    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" event={"ID":"ed916505-a5f7-4764-8fca-68f8b057337d","Type":"ContainerDied","Data":"4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553"}
May 25 13:48:13.676604 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:13.676569    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" event={"ID":"ed916505-a5f7-4764-8fca-68f8b057337d","Type":"ContainerStarted","Data":"fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d"}
May 25 13:48:13.697622 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:13.697579    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podStartSLOduration=5.697539884 podStartE2EDuration="5.697539884s" podCreationTimestamp="2026-05-25 13:48:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:48:13.695151941 +0000 UTC m=+2560.956754773" watchObservedRunningTime="2026-05-25 13:48:13.697539884 +0000 UTC m=+2560.959142707"
May 25 13:48:15.853292 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.853264    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5474d9f6f4-lxtw9_cc009044-fa41-44f2-9280-f0074f407cd7/main/0.log"
May 25 13:48:15.853751 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.853733    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:48:15.971135 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.971099    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-tmp-dir\") pod \"cc009044-fa41-44f2-9280-f0074f407cd7\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") "
May 25 13:48:15.971333 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.971160    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc009044-fa41-44f2-9280-f0074f407cd7-tls-certs\") pod \"cc009044-fa41-44f2-9280-f0074f407cd7\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") "
May 25 13:48:15.971333 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.971187    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-dshm\") pod \"cc009044-fa41-44f2-9280-f0074f407cd7\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") "
May 25 13:48:15.971333 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.971317    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-kserve-provision-location\") pod \"cc009044-fa41-44f2-9280-f0074f407cd7\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") "
May 25 13:48:15.971597 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.971349    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-model-cache\") pod \"cc009044-fa41-44f2-9280-f0074f407cd7\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") "
May 25 13:48:15.971597 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.971381    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-home\") pod \"cc009044-fa41-44f2-9280-f0074f407cd7\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") "
May 25 13:48:15.971597 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.971410    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfnzp\" (UniqueName: \"kubernetes.io/projected/cc009044-fa41-44f2-9280-f0074f407cd7-kube-api-access-bfnzp\") pod \"cc009044-fa41-44f2-9280-f0074f407cd7\" (UID: \"cc009044-fa41-44f2-9280-f0074f407cd7\") "
May 25 13:48:15.971597 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.971577    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc009044-fa41-44f2-9280-f0074f407cd7" (UID: "cc009044-fa41-44f2-9280-f0074f407cd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:48:15.971816 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.971709    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:48:15.972303 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.972274    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-home" (OuterVolumeSpecName: "home") pod "cc009044-fa41-44f2-9280-f0074f407cd7" (UID: "cc009044-fa41-44f2-9280-f0074f407cd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:48:15.973645 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.973617    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc009044-fa41-44f2-9280-f0074f407cd7-kube-api-access-bfnzp" (OuterVolumeSpecName: "kube-api-access-bfnzp") pod "cc009044-fa41-44f2-9280-f0074f407cd7" (UID: "cc009044-fa41-44f2-9280-f0074f407cd7"). InnerVolumeSpecName "kube-api-access-bfnzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:48:15.973834 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.973814    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc009044-fa41-44f2-9280-f0074f407cd7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc009044-fa41-44f2-9280-f0074f407cd7" (UID: "cc009044-fa41-44f2-9280-f0074f407cd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:48:15.973914 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.973867    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-dshm" (OuterVolumeSpecName: "dshm") pod "cc009044-fa41-44f2-9280-f0074f407cd7" (UID: "cc009044-fa41-44f2-9280-f0074f407cd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:48:15.986945 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:15.986920    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc009044-fa41-44f2-9280-f0074f407cd7" (UID: "cc009044-fa41-44f2-9280-f0074f407cd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:48:16.032486 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.032459    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc009044-fa41-44f2-9280-f0074f407cd7" (UID: "cc009044-fa41-44f2-9280-f0074f407cd7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:48:16.072706 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.072681    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc009044-fa41-44f2-9280-f0074f407cd7-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:48:16.072706 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.072703    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:48:16.072836 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.072712    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:48:16.072836 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.072721    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:48:16.072836 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.072730    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfnzp\" (UniqueName: \"kubernetes.io/projected/cc009044-fa41-44f2-9280-f0074f407cd7-kube-api-access-bfnzp\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:48:16.072836 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.072738    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc009044-fa41-44f2-9280-f0074f407cd7-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:48:16.688412 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.688385    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5474d9f6f4-lxtw9_cc009044-fa41-44f2-9280-f0074f407cd7/main/0.log"
May 25 13:48:16.688741 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.688721    2573 generic.go:358] "Generic (PLEG): container finished" podID="cc009044-fa41-44f2-9280-f0074f407cd7" containerID="e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13" exitCode=137
May 25 13:48:16.688812 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.688763    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" event={"ID":"cc009044-fa41-44f2-9280-f0074f407cd7","Type":"ContainerDied","Data":"e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13"}
May 25 13:48:16.688812 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.688784    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9" event={"ID":"cc009044-fa41-44f2-9280-f0074f407cd7","Type":"ContainerDied","Data":"417cbe494cbe5b54a78709617db350483651aa0b4dd42a8ca591359ce10c7fd5"}
May 25 13:48:16.688812 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.688792    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"
May 25 13:48:16.688987 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.688798    2573 scope.go:117] "RemoveContainer" containerID="e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13"
May 25 13:48:16.699461 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.699444    2573 scope.go:117] "RemoveContainer" containerID="386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668"
May 25 13:48:16.714456 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.714430    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"]
May 25 13:48:16.717428 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.717408    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-lxtw9"]
May 25 13:48:16.717501 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.717461    2573 scope.go:117] "RemoveContainer" containerID="e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13"
May 25 13:48:16.717813 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:48:16.717776    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13\": container with ID starting with e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13 not found: ID does not exist" containerID="e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13"
May 25 13:48:16.717891 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.717824    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13"} err="failed to get container status \"e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13\": rpc error: code = NotFound desc = could not find container \"e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13\": container with ID starting with e5af4715f4c644ccf7eb168e1b229d4b67d6b8a5b0ec2a842cd6abb0bed3ea13 not found: ID does not exist"
May 25 13:48:16.717891 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.717847    2573 scope.go:117] "RemoveContainer" containerID="386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668"
May 25 13:48:16.718120 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:48:16.718103    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668\": container with ID starting with 386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668 not found: ID does not exist" containerID="386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668"
May 25 13:48:16.718176 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:16.718125    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668"} err="failed to get container status \"386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668\": rpc error: code = NotFound desc = could not find container \"386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668\": container with ID starting with 386bde7e21b420af4100bf2e0bb86793bfb4255d7b323110760e301f16a67668 not found: ID does not exist"
May 25 13:48:17.337852 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:17.337815    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" path="/var/lib/kubelet/pods/cc009044-fa41-44f2-9280-f0074f407cd7/volumes"
May 25 13:48:18.346694 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:18.346657    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:18.347128 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:18.346702    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:48:18.348206 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:18.348179    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 13:48:28.347034 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:28.346994    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 13:48:38.347058 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:38.347019    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 13:48:48.346805 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:48.346766    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 13:48:58.346678 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:48:58.346627    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 13:49:08.347362 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:49:08.347307    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 13:49:18.347001 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:49:18.346961    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 13:49:28.346950 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:49:28.346905    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 13:49:38.347128 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:49:38.347080    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 13:49:48.356954 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:49:48.356916    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:49:48.364691 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:49:48.364668    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:49:49.364819 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:49:49.364785    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"]
May 25 13:49:50.002589 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:49:50.002506    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main" containerID="cri-o://fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d" gracePeriod=30
May 25 13:50:20.255911 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.255860    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5474d9f6f4-kj84v_ed916505-a5f7-4764-8fca-68f8b057337d/main/0.log"
May 25 13:50:20.256258 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.256239    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:50:20.346902 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.346878    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-model-cache\") pod \"ed916505-a5f7-4764-8fca-68f8b057337d\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") "
May 25 13:50:20.347021 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.346909    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-kserve-provision-location\") pod \"ed916505-a5f7-4764-8fca-68f8b057337d\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") "
May 25 13:50:20.347021 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.346942    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-dshm\") pod \"ed916505-a5f7-4764-8fca-68f8b057337d\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") "
May 25 13:50:20.347021 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.346960    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-tmp-dir\") pod \"ed916505-a5f7-4764-8fca-68f8b057337d\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") "
May 25 13:50:20.347021 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.346991    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9ghj\" (UniqueName: \"kubernetes.io/projected/ed916505-a5f7-4764-8fca-68f8b057337d-kube-api-access-v9ghj\") pod \"ed916505-a5f7-4764-8fca-68f8b057337d\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") "
May 25 13:50:20.347021 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.347018    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed916505-a5f7-4764-8fca-68f8b057337d-tls-certs\") pod \"ed916505-a5f7-4764-8fca-68f8b057337d\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") "
May 25 13:50:20.347297 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.347176    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed916505-a5f7-4764-8fca-68f8b057337d" (UID: "ed916505-a5f7-4764-8fca-68f8b057337d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:50:20.347297 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.347057    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-home\") pod \"ed916505-a5f7-4764-8fca-68f8b057337d\" (UID: \"ed916505-a5f7-4764-8fca-68f8b057337d\") "
May 25 13:50:20.347541 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.347511    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:50:20.347696 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.347679    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-home" (OuterVolumeSpecName: "home") pod "ed916505-a5f7-4764-8fca-68f8b057337d" (UID: "ed916505-a5f7-4764-8fca-68f8b057337d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:50:20.349204 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.349181    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-dshm" (OuterVolumeSpecName: "dshm") pod "ed916505-a5f7-4764-8fca-68f8b057337d" (UID: "ed916505-a5f7-4764-8fca-68f8b057337d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:50:20.349315 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.349193    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed916505-a5f7-4764-8fca-68f8b057337d-kube-api-access-v9ghj" (OuterVolumeSpecName: "kube-api-access-v9ghj") pod "ed916505-a5f7-4764-8fca-68f8b057337d" (UID: "ed916505-a5f7-4764-8fca-68f8b057337d"). InnerVolumeSpecName "kube-api-access-v9ghj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:50:20.349620 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.349599    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed916505-a5f7-4764-8fca-68f8b057337d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed916505-a5f7-4764-8fca-68f8b057337d" (UID: "ed916505-a5f7-4764-8fca-68f8b057337d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:50:20.360286 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.360262    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed916505-a5f7-4764-8fca-68f8b057337d" (UID: "ed916505-a5f7-4764-8fca-68f8b057337d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:50:20.402648 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.402615    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed916505-a5f7-4764-8fca-68f8b057337d" (UID: "ed916505-a5f7-4764-8fca-68f8b057337d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:50:20.448038 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.448017    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:50:20.448038 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.448037    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:50:20.448159 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.448047    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:50:20.448159 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.448056    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed916505-a5f7-4764-8fca-68f8b057337d-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:50:20.448159 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.448066    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9ghj\" (UniqueName: \"kubernetes.io/projected/ed916505-a5f7-4764-8fca-68f8b057337d-kube-api-access-v9ghj\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:50:20.448159 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:20.448074    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed916505-a5f7-4764-8fca-68f8b057337d-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:50:21.101123 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.101099    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5474d9f6f4-kj84v_ed916505-a5f7-4764-8fca-68f8b057337d/main/0.log"
May 25 13:50:21.101473 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.101452    2573 generic.go:358] "Generic (PLEG): container finished" podID="ed916505-a5f7-4764-8fca-68f8b057337d" containerID="fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d" exitCode=137
May 25 13:50:21.101524 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.101516    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" event={"ID":"ed916505-a5f7-4764-8fca-68f8b057337d","Type":"ContainerDied","Data":"fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d"}
May 25 13:50:21.101593 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.101529    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"
May 25 13:50:21.101593 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.101563    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v" event={"ID":"ed916505-a5f7-4764-8fca-68f8b057337d","Type":"ContainerDied","Data":"46026ec562809c602ded7bdc770397e141594e3094476abaa6949185251d2ba4"}
May 25 13:50:21.101593 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.101582    2573 scope.go:117] "RemoveContainer" containerID="fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d"
May 25 13:50:21.110408 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.110385    2573 scope.go:117] "RemoveContainer" containerID="4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553"
May 25 13:50:21.123636 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.123609    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"]
May 25 13:50:21.126459 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.126444    2573 scope.go:117] "RemoveContainer" containerID="fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d"
May 25 13:50:21.126829 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:50:21.126717    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d\": container with ID starting with fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d not found: ID does not exist" containerID="fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d"
May 25 13:50:21.126829 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.126755    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d"} err="failed to get container status \"fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d\": rpc error: code = NotFound desc = could not find container \"fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d\": container with ID starting with fbb474e3cfb7fea8e6a8082008fe4dfef0a0377c7d7cd96dd6b188b886f7801d not found: ID does not exist"
May 25 13:50:21.126829 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.126779    2573 scope.go:117] "RemoveContainer" containerID="4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553"
May 25 13:50:21.127088 ip-10-0-135-71 kubenswrapper[2573]: E0525 13:50:21.127069    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553\": container with ID starting with 4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553 not found: ID does not exist" containerID="4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553"
May 25 13:50:21.127158 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.127094    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553"} err="failed to get container status \"4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553\": rpc error: code = NotFound desc = could not find container \"4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553\": container with ID starting with 4e415b3d2d904cfcd94077323cc82bdc993b190a9a4e46c9ebad51afea64a553 not found: ID does not exist"
May 25 13:50:21.127218 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.127187    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kj84v"]
May 25 13:50:21.338190 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:21.338164    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" path="/var/lib/kubelet/pods/ed916505-a5f7-4764-8fca-68f8b057337d/volumes"
May 25 13:50:33.434762 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:33.434735    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:50:33.439665 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:50:33.439641    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:51:13.770250 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770218    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"]
May 25 13:51:13.770876 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770708    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="storage-initializer"
May 25 13:51:13.770876 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770731    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="storage-initializer"
May 25 13:51:13.770876 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770755    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="storage-initializer"
May 25 13:51:13.770876 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770765    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="storage-initializer"
May 25 13:51:13.770876 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770774    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main"
May 25 13:51:13.770876 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770783    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main"
May 25 13:51:13.770876 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770811    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main"
May 25 13:51:13.770876 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770819    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main"
May 25 13:51:13.771284 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770914    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="cc009044-fa41-44f2-9280-f0074f407cd7" containerName="main"
May 25 13:51:13.771284 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.770933    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed916505-a5f7-4764-8fca-68f8b057337d" containerName="main"
May 25 13:51:13.774023 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.774003    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:13.776220 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.776201    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
May 25 13:51:13.785446 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.785423    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"]
May 25 13:51:13.929104 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.929076    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a896e5c1-f7e9-4475-95d3-e60edb126d11-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:13.929207 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.929106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:13.929207 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.929129    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:13.929207 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.929151    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:13.929207 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.929181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:13.929350 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.929225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:13.929350 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:13.929252    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpxz\" (UniqueName: \"kubernetes.io/projected/a896e5c1-f7e9-4475-95d3-e60edb126d11-kube-api-access-nrpxz\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030120 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030054    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030120 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030092    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpxz\" (UniqueName: \"kubernetes.io/projected/a896e5c1-f7e9-4475-95d3-e60edb126d11-kube-api-access-nrpxz\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030338 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030131    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a896e5c1-f7e9-4475-95d3-e60edb126d11-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030338 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030148    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030338 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030170    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030338 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030187    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030338 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030210    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030629 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030468    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030629 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030508    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030698 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030652    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.030698 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.030669    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.032657 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.032638    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.032715 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.032703    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a896e5c1-f7e9-4475-95d3-e60edb126d11-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.040906 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.040886    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpxz\" (UniqueName: \"kubernetes.io/projected/a896e5c1-f7e9-4475-95d3-e60edb126d11-kube-api-access-nrpxz\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.086812 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.086791    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 13:51:14.215302 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.215278    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"]
May 25 13:51:14.217184 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:51:14.217159    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda896e5c1_f7e9_4475_95d3_e60edb126d11.slice/crio-5f8e933aa377abfacd49619316efea07e8293693386fd899c4c428825d73b548 WatchSource:0}: Error finding container 5f8e933aa377abfacd49619316efea07e8293693386fd899c4c428825d73b548: Status 404 returned error can't find the container with id 5f8e933aa377abfacd49619316efea07e8293693386fd899c4c428825d73b548
May 25 13:51:14.274887 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:14.274857    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk" event={"ID":"a896e5c1-f7e9-4475-95d3-e60edb126d11","Type":"ContainerStarted","Data":"5f8e933aa377abfacd49619316efea07e8293693386fd899c4c428825d73b548"}
May 25 13:51:15.280245 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:51:15.280202    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk" event={"ID":"a896e5c1-f7e9-4475-95d3-e60edb126d11","Type":"ContainerStarted","Data":"66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519"}
May 25 13:55:33.456256 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:55:33.456227    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:55:33.461563 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:55:33.461521    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 13:58:46.624938 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:46.624907    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"]
May 25 13:58:46.627306 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:46.625242    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="llm-d-routing-sidecar" containerID="cri-o://3744a7f306fbc719d21e94ddbd120f2823ac066bd66b21292e9374c6b6ddc8f8" gracePeriod=30
May 25 13:58:46.627306 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:46.625318    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="storage-initializer" containerID="cri-o://c040fa1b8475018960685e8aa72c0eae1632909cf827369bb3fcf61930a5f342" gracePeriod=30
May 25 13:58:46.630665 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:46.630640    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"]
May 25 13:58:46.630982 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:46.630960    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main" containerID="cri-o://d3469d915c434d6cd7260fbaacee0e98c71e365a21a0ffcb84b45b2aecf44425" gracePeriod=30
May 25 13:58:47.712253 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:47.712217    2573 generic.go:358] "Generic (PLEG): container finished" podID="a054017e-3e80-42f6-b448-315eb7107638" containerID="3744a7f306fbc719d21e94ddbd120f2823ac066bd66b21292e9374c6b6ddc8f8" exitCode=0
May 25 13:58:47.712751 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:47.712289    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" event={"ID":"a054017e-3e80-42f6-b448-315eb7107638","Type":"ContainerDied","Data":"3744a7f306fbc719d21e94ddbd120f2823ac066bd66b21292e9374c6b6ddc8f8"}
May 25 13:58:48.889398 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:48.889359    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 25 13:58:51.765376 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:51.765338    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 25 13:58:55.103540 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.103506    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"]
May 25 13:58:55.107022 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.107002    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.109391 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.109365    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-cp52d\""
May 25 13:58:55.109525 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.109386    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 25 13:58:55.114191 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.114170    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"]
May 25 13:58:55.117403 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.117383    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.120032 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.120008    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"]
May 25 13:58:55.134824 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.134801    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"]
May 25 13:58:55.175805 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.175776    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-model-cache\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.175805 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.175805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8z92\" (UniqueName: \"kubernetes.io/projected/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kube-api-access-c8z92\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.175947 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.175822    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.175947 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.175842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.175947 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.175862    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.175947 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.175877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.175947 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.175914    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.176109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.175962    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-home\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.176109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.175989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-dshm\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.176109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.176027    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.176109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.176048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tls-certs\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.176109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.176073    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wxq\" (UniqueName: \"kubernetes.io/projected/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kube-api-access-p9wxq\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.176109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.176088    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.176109 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.176106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-home\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.276663 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276639    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-home\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.276783 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276665    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-dshm\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.276783 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276691    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.276905 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276796    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tls-certs\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.276905 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276842    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wxq\" (UniqueName: \"kubernetes.io/projected/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kube-api-access-p9wxq\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.276905 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276868    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.276905 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276890    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-home\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.277119 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276916    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-model-cache\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.277119 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276939    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c8z92\" (UniqueName: \"kubernetes.io/projected/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kube-api-access-c8z92\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.277119 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276962    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.277119 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.276997    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.277119 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277027    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.277119 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277051    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.277119 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277101    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.277119 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277109    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-home\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.277531 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277167    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.277531 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277433    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.277531 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277473    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-home\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.277734 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277541    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.277734 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277640    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.277807 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277737    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-model-cache\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.277850 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.277830    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.279483 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.279459    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-dshm\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.279483 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.279474    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.279608 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.279484    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.279696 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.279682    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tls-certs\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.286676 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.286656    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wxq\" (UniqueName: \"kubernetes.io/projected/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kube-api-access-p9wxq\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.286755 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.286713    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8z92\" (UniqueName: \"kubernetes.io/projected/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kube-api-access-c8z92\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.416720 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.416666    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:55.429539 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.429520    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:58:55.556497 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.556472    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"]
May 25 13:58:55.559986 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:58:55.559958    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fbbd50_ff37_43c4_86c7_f20b2c3aa994.slice/crio-36844c2d356df3c7bc7ee97a963cfea2085c523f08ce1c8d6020b9553b36dad1 WatchSource:0}: Error finding container 36844c2d356df3c7bc7ee97a963cfea2085c523f08ce1c8d6020b9553b36dad1: Status 404 returned error can't find the container with id 36844c2d356df3c7bc7ee97a963cfea2085c523f08ce1c8d6020b9553b36dad1
May 25 13:58:55.562418 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.562402    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:58:55.580059 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.579953    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"]
May 25 13:58:55.582657 ip-10-0-135-71 kubenswrapper[2573]: W0525 13:58:55.582635    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7375c0fe_082b_4b83_98f8_ec6ddfced4d2.slice/crio-8a4e63ea8b510e085c28008fcce851dff7af550d0f491690045d0243111c149e WatchSource:0}: Error finding container 8a4e63ea8b510e085c28008fcce851dff7af550d0f491690045d0243111c149e: Status 404 returned error can't find the container with id 8a4e63ea8b510e085c28008fcce851dff7af550d0f491690045d0243111c149e
May 25 13:58:55.739765 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.739728    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" event={"ID":"7375c0fe-082b-4b83-98f8-ec6ddfced4d2","Type":"ContainerStarted","Data":"6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7"}
May 25 13:58:55.739944 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.739777    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" event={"ID":"7375c0fe-082b-4b83-98f8-ec6ddfced4d2","Type":"ContainerStarted","Data":"8a4e63ea8b510e085c28008fcce851dff7af550d0f491690045d0243111c149e"}
May 25 13:58:55.741144 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.741122    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" event={"ID":"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994","Type":"ContainerStarted","Data":"f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c"}
May 25 13:58:55.741249 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.741149    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" event={"ID":"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994","Type":"ContainerStarted","Data":"36844c2d356df3c7bc7ee97a963cfea2085c523f08ce1c8d6020b9553b36dad1"}
May 25 13:58:55.741297 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:55.741272    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:58:56.747243 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:56.747200    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" event={"ID":"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994","Type":"ContainerStarted","Data":"c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0"}
May 25 13:58:58.890196 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:58:58.890150    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 25 13:59:01.765471 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:01.765430    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 25 13:59:07.762248 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:07.762209    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:59:08.889920 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:08.889874    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 25 13:59:08.890302 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:08.889961    2573 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:59:11.765380 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:11.765335    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 25 13:59:12.801506 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:12.801474    2573 generic.go:358] "Generic (PLEG): container finished" podID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerID="6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7" exitCode=0
May 25 13:59:12.801860 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:12.801563    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" event={"ID":"7375c0fe-082b-4b83-98f8-ec6ddfced4d2","Type":"ContainerDied","Data":"6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7"}
May 25 13:59:13.806323 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:13.806290    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" event={"ID":"7375c0fe-082b-4b83-98f8-ec6ddfced4d2","Type":"ContainerStarted","Data":"7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9"}
May 25 13:59:13.826805 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:13.826763    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podStartSLOduration=18.826749162 podStartE2EDuration="18.826749162s" podCreationTimestamp="2026-05-25 13:58:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:59:13.824501612 +0000 UTC m=+3221.086104446" watchObservedRunningTime="2026-05-25 13:59:13.826749162 +0000 UTC m=+3221.088352011"
May 25 13:59:15.429979 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:15.429906    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:59:15.429979 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:15.429945    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 13:59:15.431504 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:15.431480    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 13:59:16.822714 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.822681    2573 generic.go:358] "Generic (PLEG): container finished" podID="df534de3-b090-4761-a621-c52543056dc0" containerID="d3469d915c434d6cd7260fbaacee0e98c71e365a21a0ffcb84b45b2aecf44425" exitCode=137
May 25 13:59:16.823163 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.822767    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" event={"ID":"df534de3-b090-4761-a621-c52543056dc0","Type":"ContainerDied","Data":"d3469d915c434d6cd7260fbaacee0e98c71e365a21a0ffcb84b45b2aecf44425"}
May 25 13:59:16.824419 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.824392    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx_a054017e-3e80-42f6-b448-315eb7107638/storage-initializer/0.log"
May 25 13:59:16.824785 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.824764    2573 generic.go:358] "Generic (PLEG): container finished" podID="a054017e-3e80-42f6-b448-315eb7107638" containerID="c040fa1b8475018960685e8aa72c0eae1632909cf827369bb3fcf61930a5f342" exitCode=137
May 25 13:59:16.824941 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.824801    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" event={"ID":"a054017e-3e80-42f6-b448-315eb7107638","Type":"ContainerDied","Data":"c040fa1b8475018960685e8aa72c0eae1632909cf827369bb3fcf61930a5f342"}
May 25 13:59:16.844440 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.844420    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx_a054017e-3e80-42f6-b448-315eb7107638/storage-initializer/0.log"
May 25 13:59:16.844872 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.844842    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:59:16.922575 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.922536    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:59:16.961121 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961095    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-dshm\") pod \"a054017e-3e80-42f6-b448-315eb7107638\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") "
May 25 13:59:16.961253 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961155    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-home\") pod \"a054017e-3e80-42f6-b448-315eb7107638\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") "
May 25 13:59:16.961253 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961194    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-tmp-dir\") pod \"a054017e-3e80-42f6-b448-315eb7107638\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") "
May 25 13:59:16.961253 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961242    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpxw7\" (UniqueName: \"kubernetes.io/projected/a054017e-3e80-42f6-b448-315eb7107638-kube-api-access-bpxw7\") pod \"a054017e-3e80-42f6-b448-315eb7107638\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") "
May 25 13:59:16.961422 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961280    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a054017e-3e80-42f6-b448-315eb7107638-tls-certs\") pod \"a054017e-3e80-42f6-b448-315eb7107638\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") "
May 25 13:59:16.961422 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961309    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-model-cache\") pod \"a054017e-3e80-42f6-b448-315eb7107638\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") "
May 25 13:59:16.961422 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961393    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-kserve-provision-location\") pod \"a054017e-3e80-42f6-b448-315eb7107638\" (UID: \"a054017e-3e80-42f6-b448-315eb7107638\") "
May 25 13:59:16.961617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961414    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-home" (OuterVolumeSpecName: "home") pod "a054017e-3e80-42f6-b448-315eb7107638" (UID: "a054017e-3e80-42f6-b448-315eb7107638"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:16.961617 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961427    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a054017e-3e80-42f6-b448-315eb7107638" (UID: "a054017e-3e80-42f6-b448-315eb7107638"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:16.961737 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961630    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-model-cache" (OuterVolumeSpecName: "model-cache") pod "a054017e-3e80-42f6-b448-315eb7107638" (UID: "a054017e-3e80-42f6-b448-315eb7107638"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:16.961737 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961660    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:16.961737 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.961679    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:16.963523 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.963499    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-dshm" (OuterVolumeSpecName: "dshm") pod "a054017e-3e80-42f6-b448-315eb7107638" (UID: "a054017e-3e80-42f6-b448-315eb7107638"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:16.963842 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.963825    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a054017e-3e80-42f6-b448-315eb7107638-kube-api-access-bpxw7" (OuterVolumeSpecName: "kube-api-access-bpxw7") pod "a054017e-3e80-42f6-b448-315eb7107638" (UID: "a054017e-3e80-42f6-b448-315eb7107638"). InnerVolumeSpecName "kube-api-access-bpxw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:59:16.963913 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.963892    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a054017e-3e80-42f6-b448-315eb7107638-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a054017e-3e80-42f6-b448-315eb7107638" (UID: "a054017e-3e80-42f6-b448-315eb7107638"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:59:16.981644 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:16.981609    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a054017e-3e80-42f6-b448-315eb7107638" (UID: "a054017e-3e80-42f6-b448-315eb7107638"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:17.062895 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.062870    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-model-cache\") pod \"df534de3-b090-4761-a621-c52543056dc0\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") "
May 25 13:59:17.063009 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.062913    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gpp5\" (UniqueName: \"kubernetes.io/projected/df534de3-b090-4761-a621-c52543056dc0-kube-api-access-4gpp5\") pod \"df534de3-b090-4761-a621-c52543056dc0\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") "
May 25 13:59:17.063009 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.062931    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df534de3-b090-4761-a621-c52543056dc0-tls-certs\") pod \"df534de3-b090-4761-a621-c52543056dc0\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") "
May 25 13:59:17.063009 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.062950    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-tmp-dir\") pod \"df534de3-b090-4761-a621-c52543056dc0\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") "
May 25 13:59:17.063009 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.062971    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-kserve-provision-location\") pod \"df534de3-b090-4761-a621-c52543056dc0\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") "
May 25 13:59:17.063238 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.063127    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "df534de3-b090-4761-a621-c52543056dc0" (UID: "df534de3-b090-4761-a621-c52543056dc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:17.063238 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.063235    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-home\") pod \"df534de3-b090-4761-a621-c52543056dc0\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") "
May 25 13:59:17.063341 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.063319    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-dshm\") pod \"df534de3-b090-4761-a621-c52543056dc0\" (UID: \"df534de3-b090-4761-a621-c52543056dc0\") "
May 25 13:59:17.063646 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.063607    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpxw7\" (UniqueName: \"kubernetes.io/projected/a054017e-3e80-42f6-b448-315eb7107638-kube-api-access-bpxw7\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.063646 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.063631    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a054017e-3e80-42f6-b448-315eb7107638-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.063646 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.063647    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.063840 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.063660    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.063840 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.063673    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.063840 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.063685    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a054017e-3e80-42f6-b448-315eb7107638-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.063962 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.063836    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-home" (OuterVolumeSpecName: "home") pod "df534de3-b090-4761-a621-c52543056dc0" (UID: "df534de3-b090-4761-a621-c52543056dc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:17.065407 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.065376    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df534de3-b090-4761-a621-c52543056dc0-kube-api-access-4gpp5" (OuterVolumeSpecName: "kube-api-access-4gpp5") pod "df534de3-b090-4761-a621-c52543056dc0" (UID: "df534de3-b090-4761-a621-c52543056dc0"). InnerVolumeSpecName "kube-api-access-4gpp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:59:17.065726 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.065706    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df534de3-b090-4761-a621-c52543056dc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df534de3-b090-4761-a621-c52543056dc0" (UID: "df534de3-b090-4761-a621-c52543056dc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:59:17.065893 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.065866    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-dshm" (OuterVolumeSpecName: "dshm") pod "df534de3-b090-4761-a621-c52543056dc0" (UID: "df534de3-b090-4761-a621-c52543056dc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:17.076028 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.076004    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df534de3-b090-4761-a621-c52543056dc0" (UID: "df534de3-b090-4761-a621-c52543056dc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:17.132808 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.132782    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df534de3-b090-4761-a621-c52543056dc0" (UID: "df534de3-b090-4761-a621-c52543056dc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:17.164885 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.164811    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.164885 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.164833    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gpp5\" (UniqueName: \"kubernetes.io/projected/df534de3-b090-4761-a621-c52543056dc0-kube-api-access-4gpp5\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.164885 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.164845    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df534de3-b090-4761-a621-c52543056dc0-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.164885 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.164862    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.164885 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.164871    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.164885 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.164879    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df534de3-b090-4761-a621-c52543056dc0-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 13:59:17.829165 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.829127    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7" event={"ID":"df534de3-b090-4761-a621-c52543056dc0","Type":"ContainerDied","Data":"2cc785ad2a7cb8f9e5c2c75d9264dea41b570a8fabecf1ccb5d32a9e26366069"}
May 25 13:59:17.829655 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.829183    2573 scope.go:117] "RemoveContainer" containerID="d3469d915c434d6cd7260fbaacee0e98c71e365a21a0ffcb84b45b2aecf44425"
May 25 13:59:17.829655 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.829141    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"
May 25 13:59:17.830516 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.830295    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx_a054017e-3e80-42f6-b448-315eb7107638/storage-initializer/0.log"
May 25 13:59:17.830772 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.830727    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx" event={"ID":"a054017e-3e80-42f6-b448-315eb7107638","Type":"ContainerDied","Data":"0afa0717f45efe35bb1f7bcd14e0b5076f69e8515b103f54a80226a6b477e0ae"}
May 25 13:59:17.830843 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.830787    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"
May 25 13:59:17.838034 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.838020    2573 scope.go:117] "RemoveContainer" containerID="613bba6d633313b9346c5affd2b8c39be353ef2cc974037e5c6477144b15139c"
May 25 13:59:17.865557 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.865519    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"]
May 25 13:59:17.871128 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.871108    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-sxbwx"]
May 25 13:59:17.882606 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.882528    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"]
May 25 13:59:17.884569 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.884533    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-fh2q7"]
May 25 13:59:17.903333 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.903315    2573 scope.go:117] "RemoveContainer" containerID="c040fa1b8475018960685e8aa72c0eae1632909cf827369bb3fcf61930a5f342"
May 25 13:59:17.932864 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:17.932832    2573 scope.go:117] "RemoveContainer" containerID="3744a7f306fbc719d21e94ddbd120f2823ac066bd66b21292e9374c6b6ddc8f8"
May 25 13:59:19.337384 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:19.337340    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a054017e-3e80-42f6-b448-315eb7107638" path="/var/lib/kubelet/pods/a054017e-3e80-42f6-b448-315eb7107638/volumes"
May 25 13:59:19.337910 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:19.337780    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df534de3-b090-4761-a621-c52543056dc0" path="/var/lib/kubelet/pods/df534de3-b090-4761-a621-c52543056dc0/volumes"
May 25 13:59:25.430128 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:25.430084    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 13:59:35.429951 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:35.429906    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 13:59:45.430808 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:45.430764    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 13:59:49.939178 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:49.939143    2573 generic.go:358] "Generic (PLEG): container finished" podID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerID="c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0" exitCode=0
May 25 13:59:49.939684 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:49.939217    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" event={"ID":"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994","Type":"ContainerDied","Data":"c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0"}
May 25 13:59:50.944613 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:50.944582    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" event={"ID":"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994","Type":"ContainerStarted","Data":"0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417"}
May 25 13:59:50.967399 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:50.967359    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podStartSLOduration=55.967342179 podStartE2EDuration="55.967342179s" podCreationTimestamp="2026-05-25 13:58:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:59:50.96608416 +0000 UTC m=+3258.227686990" watchObservedRunningTime="2026-05-25 13:59:50.967342179 +0000 UTC m=+3258.228945001"
May 25 13:59:55.417081 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:55.417046    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:59:55.417081 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:55.417085    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 13:59:55.417627 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:55.417427    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8001/health\": dial tcp 10.132.0.41:8001: connect: connection refused"
May 25 13:59:55.430327 ip-10-0-135-71 kubenswrapper[2573]: I0525 13:59:55.430296    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 14:00:05.417876 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:05.417826    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8001/health\": dial tcp 10.132.0.41:8001: connect: connection refused"
May 25 14:00:05.430208 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:05.430159    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 14:00:15.417518 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:15.417468    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8001/health\": dial tcp 10.132.0.41:8001: connect: connection refused"
May 25 14:00:15.429928 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:15.429890    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 14:00:25.417983 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:25.417938    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8001/health\": dial tcp 10.132.0.41:8001: connect: connection refused"
May 25 14:00:25.430375 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:25.430336    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 14:00:33.488241 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:33.488205    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 14:00:33.492700 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:33.492672    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 14:00:35.417496 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:35.417448    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8001/health\": dial tcp 10.132.0.41:8001: connect: connection refused"
May 25 14:00:35.430604 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:35.430540    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 14:00:45.417427 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:45.417321    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8001/health\": dial tcp 10.132.0.41:8001: connect: connection refused"
May 25 14:00:45.440316 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:45.440290    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 14:00:45.448265 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:45.448242    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 14:00:55.417555 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:00:55.417488    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8001/health\": dial tcp 10.132.0.41:8001: connect: connection refused"
May 25 14:01:05.417421 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:01:05.417367    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8001/health\": dial tcp 10.132.0.41:8001: connect: connection refused"
May 25 14:01:15.417104 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:01:15.417050    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8001/health\": dial tcp 10.132.0.41:8001: connect: connection refused"
May 25 14:01:25.427252 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:01:25.427223    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 14:01:25.439747 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:01:25.439725    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 14:01:36.306412 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:01:36.306358    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"]
May 25 14:01:36.307202 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:01:36.307147    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main" containerID="cri-o://0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417" gracePeriod=30
May 25 14:01:36.308536 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:01:36.308514    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"]
May 25 14:01:36.308903 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:01:36.308845    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main" containerID="cri-o://7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9" gracePeriod=30
May 25 14:02:06.307376 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.307337    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="llm-d-routing-sidecar" containerID="cri-o://f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c" gracePeriod=2
May 25 14:02:06.601880 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.601859    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 14:02:06.604729 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.604711    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k_e1fbbd50-ff37-43c4-86c7-f20b2c3aa994/main/0.log"
May 25 14:02:06.605443 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.605427    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 14:02:06.776577 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.776532    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kserve-provision-location\") pod \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") "
May 25 14:02:06.776694 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.776591    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9wxq\" (UniqueName: \"kubernetes.io/projected/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kube-api-access-p9wxq\") pod \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") "
May 25 14:02:06.776694 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.776622    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-model-cache\") pod \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") "
May 25 14:02:06.776694 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.776638    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-dshm\") pod \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") "
May 25 14:02:06.776694 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.776655    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tmp-dir\") pod \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") "
May 25 14:02:06.776694 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.776675    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8z92\" (UniqueName: \"kubernetes.io/projected/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kube-api-access-c8z92\") pod \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") "
May 25 14:02:06.776694 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.776690    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kserve-provision-location\") pod \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") "
May 25 14:02:06.777010 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.776707    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tls-certs\") pod \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") "
May 25 14:02:06.777010 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.776871    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7375c0fe-082b-4b83-98f8-ec6ddfced4d2" (UID: "7375c0fe-082b-4b83-98f8-ec6ddfced4d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:06.777010 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.776975    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-home\") pod \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") "
May 25 14:02:06.777184 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.777012    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tmp-dir\") pod \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") "
May 25 14:02:06.777184 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.777038    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tls-certs\") pod \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") "
May 25 14:02:06.777184 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.777078    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-model-cache\") pod \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") "
May 25 14:02:06.777184 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.777108    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-dshm\") pod \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\" (UID: \"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994\") "
May 25 14:02:06.777184 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.777135    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-home\") pod \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\" (UID: \"7375c0fe-082b-4b83-98f8-ec6ddfced4d2\") "
May 25 14:02:06.777440 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.777398    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.778947 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.778092    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-home" (OuterVolumeSpecName: "home") pod "7375c0fe-082b-4b83-98f8-ec6ddfced4d2" (UID: "7375c0fe-082b-4b83-98f8-ec6ddfced4d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:06.779067 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.778991    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-home" (OuterVolumeSpecName: "home") pod "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" (UID: "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:06.779681 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.779303    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-dshm" (OuterVolumeSpecName: "dshm") pod "7375c0fe-082b-4b83-98f8-ec6ddfced4d2" (UID: "7375c0fe-082b-4b83-98f8-ec6ddfced4d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:06.779681 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.779354    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kube-api-access-p9wxq" (OuterVolumeSpecName: "kube-api-access-p9wxq") pod "7375c0fe-082b-4b83-98f8-ec6ddfced4d2" (UID: "7375c0fe-082b-4b83-98f8-ec6ddfced4d2"). InnerVolumeSpecName "kube-api-access-p9wxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:02:06.779681 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.779586    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" (UID: "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:06.780023 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.780001    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7375c0fe-082b-4b83-98f8-ec6ddfced4d2" (UID: "7375c0fe-082b-4b83-98f8-ec6ddfced4d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:02:06.780322 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.780292    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kube-api-access-c8z92" (OuterVolumeSpecName: "kube-api-access-c8z92") pod "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" (UID: "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994"). InnerVolumeSpecName "kube-api-access-c8z92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:02:06.781705 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.781620    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-dshm" (OuterVolumeSpecName: "dshm") pod "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" (UID: "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:06.781705 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.781679    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" (UID: "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:02:06.792175 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.792154    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" (UID: "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:06.793489 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.793470    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7375c0fe-082b-4b83-98f8-ec6ddfced4d2" (UID: "7375c0fe-082b-4b83-98f8-ec6ddfced4d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:06.852273 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.852223    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7375c0fe-082b-4b83-98f8-ec6ddfced4d2" (UID: "7375c0fe-082b-4b83-98f8-ec6ddfced4d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:06.852356 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.852314    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" (UID: "e1fbbd50-ff37-43c4-86c7-f20b2c3aa994"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:06.878673 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878653    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878673 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878673    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878682    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878692    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878700    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878708    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878716    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878724    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9wxq\" (UniqueName: \"kubernetes.io/projected/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-kube-api-access-p9wxq\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878733    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878740    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878748    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8z92\" (UniqueName: \"kubernetes.io/projected/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kube-api-access-c8z92\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878756    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:06.878771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:06.878765    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7375c0fe-082b-4b83-98f8-ec6ddfced4d2-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:02:07.397151 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.397131    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k_e1fbbd50-ff37-43c4-86c7-f20b2c3aa994/main/0.log"
May 25 14:02:07.397720 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.397697    2573 generic.go:358] "Generic (PLEG): container finished" podID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerID="0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417" exitCode=137
May 25 14:02:07.397720 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.397719    2573 generic.go:358] "Generic (PLEG): container finished" podID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerID="f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c" exitCode=0
May 25 14:02:07.397850 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.397763    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"
May 25 14:02:07.397850 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.397776    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" event={"ID":"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994","Type":"ContainerDied","Data":"0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417"}
May 25 14:02:07.397850 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.397809    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" event={"ID":"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994","Type":"ContainerDied","Data":"f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c"}
May 25 14:02:07.397850 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.397825    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k" event={"ID":"e1fbbd50-ff37-43c4-86c7-f20b2c3aa994","Type":"ContainerDied","Data":"36844c2d356df3c7bc7ee97a963cfea2085c523f08ce1c8d6020b9553b36dad1"}
May 25 14:02:07.398039 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.397840    2573 scope.go:117] "RemoveContainer" containerID="0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417"
May 25 14:02:07.399350 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.399325    2573 generic.go:358] "Generic (PLEG): container finished" podID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerID="7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9" exitCode=137
May 25 14:02:07.399447 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.399378    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" event={"ID":"7375c0fe-082b-4b83-98f8-ec6ddfced4d2","Type":"ContainerDied","Data":"7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9"}
May 25 14:02:07.399447 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.399401    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d" event={"ID":"7375c0fe-082b-4b83-98f8-ec6ddfced4d2","Type":"ContainerDied","Data":"8a4e63ea8b510e085c28008fcce851dff7af550d0f491690045d0243111c149e"}
May 25 14:02:07.399447 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.399405    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"
May 25 14:02:07.406152 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.406136    2573 scope.go:117] "RemoveContainer" containerID="c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0"
May 25 14:02:07.416267 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.416243    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"]
May 25 14:02:07.421520 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.421498    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-zgv2k"]
May 25 14:02:07.433260 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.433240    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"]
May 25 14:02:07.439146 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.439124    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-j4v5d"]
May 25 14:02:07.467889 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.467872    2573 scope.go:117] "RemoveContainer" containerID="f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c"
May 25 14:02:07.474692 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.474676    2573 scope.go:117] "RemoveContainer" containerID="0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417"
May 25 14:02:07.474924 ip-10-0-135-71 kubenswrapper[2573]: E0525 14:02:07.474905    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417\": container with ID starting with 0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417 not found: ID does not exist" containerID="0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417"
May 25 14:02:07.474987 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.474940    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417"} err="failed to get container status \"0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417\": rpc error: code = NotFound desc = could not find container \"0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417\": container with ID starting with 0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417 not found: ID does not exist"
May 25 14:02:07.474987 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.474957    2573 scope.go:117] "RemoveContainer" containerID="c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0"
May 25 14:02:07.475185 ip-10-0-135-71 kubenswrapper[2573]: E0525 14:02:07.475170    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0\": container with ID starting with c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0 not found: ID does not exist" containerID="c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0"
May 25 14:02:07.475225 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.475192    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0"} err="failed to get container status \"c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0\": rpc error: code = NotFound desc = could not find container \"c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0\": container with ID starting with c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0 not found: ID does not exist"
May 25 14:02:07.475225 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.475212    2573 scope.go:117] "RemoveContainer" containerID="f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c"
May 25 14:02:07.475436 ip-10-0-135-71 kubenswrapper[2573]: E0525 14:02:07.475418    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c\": container with ID starting with f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c not found: ID does not exist" containerID="f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c"
May 25 14:02:07.475499 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.475441    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c"} err="failed to get container status \"f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c\": rpc error: code = NotFound desc = could not find container \"f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c\": container with ID starting with f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c not found: ID does not exist"
May 25 14:02:07.475499 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.475456    2573 scope.go:117] "RemoveContainer" containerID="0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417"
May 25 14:02:07.475718 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.475700    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417"} err="failed to get container status \"0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417\": rpc error: code = NotFound desc = could not find container \"0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417\": container with ID starting with 0e42b5cbc5748e21cb15f7cefebaec016a78e069f13617359d4b9eec38875417 not found: ID does not exist"
May 25 14:02:07.475771 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.475718    2573 scope.go:117] "RemoveContainer" containerID="c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0"
May 25 14:02:07.475943 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.475922    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0"} err="failed to get container status \"c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0\": rpc error: code = NotFound desc = could not find container \"c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0\": container with ID starting with c4ed1e3ba98e666b9fb994bb3347eceac316239e7e41c5b3dbd38fbdc2b450b0 not found: ID does not exist"
May 25 14:02:07.475985 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.475944    2573 scope.go:117] "RemoveContainer" containerID="f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c"
May 25 14:02:07.476127 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.476112    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c"} err="failed to get container status \"f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c\": rpc error: code = NotFound desc = could not find container \"f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c\": container with ID starting with f7a13edb6b11e6f907a6914a9c9470f6ca42626b601b8e09bf211911026a701c not found: ID does not exist"
May 25 14:02:07.476174 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.476128    2573 scope.go:117] "RemoveContainer" containerID="7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9"
May 25 14:02:07.483045 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.483029    2573 scope.go:117] "RemoveContainer" containerID="6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7"
May 25 14:02:07.542666 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.542650    2573 scope.go:117] "RemoveContainer" containerID="7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9"
May 25 14:02:07.542911 ip-10-0-135-71 kubenswrapper[2573]: E0525 14:02:07.542890    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9\": container with ID starting with 7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9 not found: ID does not exist" containerID="7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9"
May 25 14:02:07.542989 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.542917    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9"} err="failed to get container status \"7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9\": rpc error: code = NotFound desc = could not find container \"7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9\": container with ID starting with 7653511e8cea1b7b93d6e23a43101e8db38b467d5089a06fc06a3fb30262b0f9 not found: ID does not exist"
May 25 14:02:07.542989 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.542930    2573 scope.go:117] "RemoveContainer" containerID="6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7"
May 25 14:02:07.543119 ip-10-0-135-71 kubenswrapper[2573]: E0525 14:02:07.543100    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7\": container with ID starting with 6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7 not found: ID does not exist" containerID="6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7"
May 25 14:02:07.543170 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:07.543128    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7"} err="failed to get container status \"6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7\": rpc error: code = NotFound desc = could not find container \"6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7\": container with ID starting with 6897cb9e5217f55789020cffc21c3137aae9c661575c92bf5993c947d65bd8c7 not found: ID does not exist"
May 25 14:02:09.337866 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:09.337833    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" path="/var/lib/kubelet/pods/7375c0fe-082b-4b83-98f8-ec6ddfced4d2/volumes"
May 25 14:02:09.338238 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:02:09.338225    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" path="/var/lib/kubelet/pods/e1fbbd50-ff37-43c4-86c7-f20b2c3aa994/volumes"
May 25 14:05:33.511636 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:05:33.511603    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 14:05:33.517936 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:05:33.517912    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 14:06:01.383682 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:01.383651    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"]
May 25 14:06:01.384157 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:01.383883    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk" podUID="a896e5c1-f7e9-4475-95d3-e60edb126d11" containerName="storage-initializer" containerID="cri-o://66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519" gracePeriod=30
May 25 14:06:11.301814 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.301777    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302057    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="storage-initializer"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302067    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="storage-initializer"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302081    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302087    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302095    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="storage-initializer"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302100    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="storage-initializer"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302105    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302110    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302116    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="llm-d-routing-sidecar"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302121    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="llm-d-routing-sidecar"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302126    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302131    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302137    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="storage-initializer"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302142    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="storage-initializer"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302148    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="storage-initializer"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302153    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="storage-initializer"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302160    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="llm-d-routing-sidecar"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302165    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="llm-d-routing-sidecar"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302210    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="storage-initializer"
May 25 14:06:11.302206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302216    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="main"
May 25 14:06:11.303000 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302224    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="7375c0fe-082b-4b83-98f8-ec6ddfced4d2" containerName="main"
May 25 14:06:11.303000 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302231    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1fbbd50-ff37-43c4-86c7-f20b2c3aa994" containerName="llm-d-routing-sidecar"
May 25 14:06:11.303000 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302238    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="df534de3-b090-4761-a621-c52543056dc0" containerName="main"
May 25 14:06:11.303000 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.302244    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="a054017e-3e80-42f6-b448-315eb7107638" containerName="llm-d-routing-sidecar"
May 25 14:06:11.306901 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.306879    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.309351 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.309321    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-x2vg8\""
May 25 14:06:11.309527 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.309500    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 25 14:06:11.326229 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.326206    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 14:06:11.451375 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.451348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4vz\" (UniqueName: \"kubernetes.io/projected/f09d7900-a2f5-4867-a5e2-37057d7ce599-kube-api-access-8z4vz\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.451514 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.451394    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.451514 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.451483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.451514 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.451510    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.451726 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.451558    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.451726 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.451590    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09d7900-a2f5-4867-a5e2-37057d7ce599-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.451726 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.451612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.552686 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.552617    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.552686 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.552672    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.552898 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.552701    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.552898 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.552724    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.552898 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.552739    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09d7900-a2f5-4867-a5e2-37057d7ce599-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.552898 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.552754    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.552898 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.552779    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4vz\" (UniqueName: \"kubernetes.io/projected/f09d7900-a2f5-4867-a5e2-37057d7ce599-kube-api-access-8z4vz\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.553167 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.553007    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.553167 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.553105    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.553240 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.553161    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.553240 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.553216    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.554955 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.554936    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.555074 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.555057    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09d7900-a2f5-4867-a5e2-37057d7ce599-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.561279 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.561260    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4vz\" (UniqueName: \"kubernetes.io/projected/f09d7900-a2f5-4867-a5e2-37057d7ce599-kube-api-access-8z4vz\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.617327 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.617306    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:06:11.742201 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.742161    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 14:06:11.746446 ip-10-0-135-71 kubenswrapper[2573]: W0525 14:06:11.746419    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09d7900_a2f5_4867_a5e2_37057d7ce599.slice/crio-c5698089c539be814e576213e4ffd583adcaf374231958e0eb87800380ad2573 WatchSource:0}: Error finding container c5698089c539be814e576213e4ffd583adcaf374231958e0eb87800380ad2573: Status 404 returned error can't find the container with id c5698089c539be814e576213e4ffd583adcaf374231958e0eb87800380ad2573
May 25 14:06:11.748480 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:11.748464    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 14:06:12.190559 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:12.190449    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"f09d7900-a2f5-4867-a5e2-37057d7ce599","Type":"ContainerStarted","Data":"04f63df08f9711b618b8ed4e68976b24f40d0e569c7cbdfe7acb3b3d184af568"}
May 25 14:06:12.190559 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:12.190488    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"f09d7900-a2f5-4867-a5e2-37057d7ce599","Type":"ContainerStarted","Data":"c5698089c539be814e576213e4ffd583adcaf374231958e0eb87800380ad2573"}
May 25 14:06:31.557438 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.557415    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk_a896e5c1-f7e9-4475-95d3-e60edb126d11/storage-initializer/0.log"
May 25 14:06:31.557841 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.557496    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 14:06:31.587256 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587226    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-dshm\") pod \"a896e5c1-f7e9-4475-95d3-e60edb126d11\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") "
May 25 14:06:31.587383 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587260    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrpxz\" (UniqueName: \"kubernetes.io/projected/a896e5c1-f7e9-4475-95d3-e60edb126d11-kube-api-access-nrpxz\") pod \"a896e5c1-f7e9-4475-95d3-e60edb126d11\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") "
May 25 14:06:31.587383 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587279    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-home\") pod \"a896e5c1-f7e9-4475-95d3-e60edb126d11\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") "
May 25 14:06:31.587383 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587294    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-kserve-provision-location\") pod \"a896e5c1-f7e9-4475-95d3-e60edb126d11\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") "
May 25 14:06:31.587383 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587324    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-model-cache\") pod \"a896e5c1-f7e9-4475-95d3-e60edb126d11\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") "
May 25 14:06:31.587383 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587354    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a896e5c1-f7e9-4475-95d3-e60edb126d11-tls-certs\") pod \"a896e5c1-f7e9-4475-95d3-e60edb126d11\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") "
May 25 14:06:31.587695 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587422    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-tmp-dir\") pod \"a896e5c1-f7e9-4475-95d3-e60edb126d11\" (UID: \"a896e5c1-f7e9-4475-95d3-e60edb126d11\") "
May 25 14:06:31.587695 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587566    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-home" (OuterVolumeSpecName: "home") pod "a896e5c1-f7e9-4475-95d3-e60edb126d11" (UID: "a896e5c1-f7e9-4475-95d3-e60edb126d11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:06:31.587695 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587610    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-model-cache" (OuterVolumeSpecName: "model-cache") pod "a896e5c1-f7e9-4475-95d3-e60edb126d11" (UID: "a896e5c1-f7e9-4475-95d3-e60edb126d11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:06:31.587862 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587747    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:06:31.587862 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587767    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:06:31.587862 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.587766    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a896e5c1-f7e9-4475-95d3-e60edb126d11" (UID: "a896e5c1-f7e9-4475-95d3-e60edb126d11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:06:31.589350 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.589320    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-dshm" (OuterVolumeSpecName: "dshm") pod "a896e5c1-f7e9-4475-95d3-e60edb126d11" (UID: "a896e5c1-f7e9-4475-95d3-e60edb126d11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:06:31.589350 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.589321    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a896e5c1-f7e9-4475-95d3-e60edb126d11-kube-api-access-nrpxz" (OuterVolumeSpecName: "kube-api-access-nrpxz") pod "a896e5c1-f7e9-4475-95d3-e60edb126d11" (UID: "a896e5c1-f7e9-4475-95d3-e60edb126d11"). InnerVolumeSpecName "kube-api-access-nrpxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:06:31.589495 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.589407    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a896e5c1-f7e9-4475-95d3-e60edb126d11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a896e5c1-f7e9-4475-95d3-e60edb126d11" (UID: "a896e5c1-f7e9-4475-95d3-e60edb126d11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:06:31.615262 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.615238    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a896e5c1-f7e9-4475-95d3-e60edb126d11" (UID: "a896e5c1-f7e9-4475-95d3-e60edb126d11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:06:31.689026 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.688973    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a896e5c1-f7e9-4475-95d3-e60edb126d11-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:06:31.689026 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.688993    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:06:31.689026 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.689001    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:06:31.689026 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.689010    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpxz\" (UniqueName: \"kubernetes.io/projected/a896e5c1-f7e9-4475-95d3-e60edb126d11-kube-api-access-nrpxz\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:06:31.689026 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:31.689019    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a896e5c1-f7e9-4475-95d3-e60edb126d11-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:06:32.256808 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:32.256784    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk_a896e5c1-f7e9-4475-95d3-e60edb126d11/storage-initializer/0.log"
May 25 14:06:32.256959 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:32.256828    2573 generic.go:358] "Generic (PLEG): container finished" podID="a896e5c1-f7e9-4475-95d3-e60edb126d11" containerID="66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519" exitCode=137
May 25 14:06:32.256959 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:32.256871    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk" event={"ID":"a896e5c1-f7e9-4475-95d3-e60edb126d11","Type":"ContainerDied","Data":"66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519"}
May 25 14:06:32.256959 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:32.256909    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk" event={"ID":"a896e5c1-f7e9-4475-95d3-e60edb126d11","Type":"ContainerDied","Data":"5f8e933aa377abfacd49619316efea07e8293693386fd899c4c428825d73b548"}
May 25 14:06:32.256959 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:32.256913    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"
May 25 14:06:32.256959 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:32.256925    2573 scope.go:117] "RemoveContainer" containerID="66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519"
May 25 14:06:32.284400 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:32.284379    2573 scope.go:117] "RemoveContainer" containerID="66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519"
May 25 14:06:32.284678 ip-10-0-135-71 kubenswrapper[2573]: E0525 14:06:32.284656    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519\": container with ID starting with 66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519 not found: ID does not exist" containerID="66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519"
May 25 14:06:32.284751 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:32.284690    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519"} err="failed to get container status \"66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519\": rpc error: code = NotFound desc = could not find container \"66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519\": container with ID starting with 66647c04c9550ed547ab579882a9dd74e50761a5b6501f40089e80762ca07519 not found: ID does not exist"
May 25 14:06:32.295878 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:32.295845    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"]
May 25 14:06:32.299032 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:32.299011    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c8vv5pk"]
May 25 14:06:33.338603 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:06:33.338571    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a896e5c1-f7e9-4475-95d3-e60edb126d11" path="/var/lib/kubelet/pods/a896e5c1-f7e9-4475-95d3-e60edb126d11/volumes"
May 25 14:07:56.533752 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:07:56.533723    2573 generic.go:358] "Generic (PLEG): container finished" podID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerID="04f63df08f9711b618b8ed4e68976b24f40d0e569c7cbdfe7acb3b3d184af568" exitCode=0
May 25 14:07:56.534066 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:07:56.533791    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"f09d7900-a2f5-4867-a5e2-37057d7ce599","Type":"ContainerDied","Data":"04f63df08f9711b618b8ed4e68976b24f40d0e569c7cbdfe7acb3b3d184af568"}
May 25 14:07:57.540094 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:07:57.540056    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"f09d7900-a2f5-4867-a5e2-37057d7ce599","Type":"ContainerStarted","Data":"2a3c4c73d8bbf3488c3b0d9d2d5f7b6148f7d3085590d3b83d8ae864e70f9e34"}
May 25 14:07:57.561480 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:07:57.561420    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=106.561404602 podStartE2EDuration="1m46.561404602s" podCreationTimestamp="2026-05-25 14:06:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:07:57.559600179 +0000 UTC m=+3744.821203009" watchObservedRunningTime="2026-05-25 14:07:57.561404602 +0000 UTC m=+3744.823007435"
May 25 14:08:01.617708 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:08:01.617676    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:08:01.618967 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:08:01.618942    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 14:08:11.617463 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:08:11.617415    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:08:11.617944 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:08:11.617745    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 14:08:21.618142 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:08:21.618059    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 14:08:31.618466 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:08:31.618423    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 14:08:41.617933 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:08:41.617891    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 14:08:51.618521 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:08:51.618480    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 14:09:01.618001 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:01.617953    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 14:09:11.618712 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:11.618661    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 14:09:21.618533 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:21.618491    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 14:09:31.627481 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:31.627452    2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:09:31.635110 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:31.635083    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:09:39.253427 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:39.253386    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 14:09:39.253845 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:39.253744    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main" containerID="cri-o://2a3c4c73d8bbf3488c3b0d9d2d5f7b6148f7d3085590d3b83d8ae864e70f9e34" gracePeriod=30
May 25 14:09:39.884689 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:39.884656    2573 generic.go:358] "Generic (PLEG): container finished" podID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerID="2a3c4c73d8bbf3488c3b0d9d2d5f7b6148f7d3085590d3b83d8ae864e70f9e34" exitCode=0
May 25 14:09:39.884810 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:39.884723    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"f09d7900-a2f5-4867-a5e2-37057d7ce599","Type":"ContainerDied","Data":"2a3c4c73d8bbf3488c3b0d9d2d5f7b6148f7d3085590d3b83d8ae864e70f9e34"}
May 25 14:09:39.912781 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:39.912761    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:09:40.001522 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.001495    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-model-cache\") pod \"f09d7900-a2f5-4867-a5e2-37057d7ce599\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") "
May 25 14:09:40.001681 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.001528    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-dshm\") pod \"f09d7900-a2f5-4867-a5e2-37057d7ce599\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") "
May 25 14:09:40.001681 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.001598    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z4vz\" (UniqueName: \"kubernetes.io/projected/f09d7900-a2f5-4867-a5e2-37057d7ce599-kube-api-access-8z4vz\") pod \"f09d7900-a2f5-4867-a5e2-37057d7ce599\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") "
May 25 14:09:40.001681 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.001624    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-kserve-provision-location\") pod \"f09d7900-a2f5-4867-a5e2-37057d7ce599\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") "
May 25 14:09:40.001681 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.001650    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09d7900-a2f5-4867-a5e2-37057d7ce599-tls-certs\") pod \"f09d7900-a2f5-4867-a5e2-37057d7ce599\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") "
May 25 14:09:40.001681 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.001679    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-home\") pod \"f09d7900-a2f5-4867-a5e2-37057d7ce599\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") "
May 25 14:09:40.001930 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.001700    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-tmp-dir\") pod \"f09d7900-a2f5-4867-a5e2-37057d7ce599\" (UID: \"f09d7900-a2f5-4867-a5e2-37057d7ce599\") "
May 25 14:09:40.001930 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.001748    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-model-cache" (OuterVolumeSpecName: "model-cache") pod "f09d7900-a2f5-4867-a5e2-37057d7ce599" (UID: "f09d7900-a2f5-4867-a5e2-37057d7ce599"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:40.001930 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.001924    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-model-cache\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:09:40.002398 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.002372    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-home" (OuterVolumeSpecName: "home") pod "f09d7900-a2f5-4867-a5e2-37057d7ce599" (UID: "f09d7900-a2f5-4867-a5e2-37057d7ce599"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:40.003804 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.003781    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-dshm" (OuterVolumeSpecName: "dshm") pod "f09d7900-a2f5-4867-a5e2-37057d7ce599" (UID: "f09d7900-a2f5-4867-a5e2-37057d7ce599"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:40.003906 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.003803    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09d7900-a2f5-4867-a5e2-37057d7ce599-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f09d7900-a2f5-4867-a5e2-37057d7ce599" (UID: "f09d7900-a2f5-4867-a5e2-37057d7ce599"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:09:40.004390 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.004368    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09d7900-a2f5-4867-a5e2-37057d7ce599-kube-api-access-8z4vz" (OuterVolumeSpecName: "kube-api-access-8z4vz") pod "f09d7900-a2f5-4867-a5e2-37057d7ce599" (UID: "f09d7900-a2f5-4867-a5e2-37057d7ce599"). InnerVolumeSpecName "kube-api-access-8z4vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:09:40.015056 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.015031    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f09d7900-a2f5-4867-a5e2-37057d7ce599" (UID: "f09d7900-a2f5-4867-a5e2-37057d7ce599"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:40.069673 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.069648    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f09d7900-a2f5-4867-a5e2-37057d7ce599" (UID: "f09d7900-a2f5-4867-a5e2-37057d7ce599"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:40.103163 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.103135    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-home\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:09:40.103163 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.103162    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-tmp-dir\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:09:40.103295 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.103176    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-dshm\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:09:40.103295 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.103190    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z4vz\" (UniqueName: \"kubernetes.io/projected/f09d7900-a2f5-4867-a5e2-37057d7ce599-kube-api-access-8z4vz\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:09:40.103295 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.103204    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f09d7900-a2f5-4867-a5e2-37057d7ce599-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:09:40.103295 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.103220    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09d7900-a2f5-4867-a5e2-37057d7ce599-tls-certs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:09:40.206897 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.206871    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6knx6/must-gather-8mgsd"]
May 25 14:09:40.207191 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.207177    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main"
May 25 14:09:40.207255 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.207193    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main"
May 25 14:09:40.207255 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.207218    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a896e5c1-f7e9-4475-95d3-e60edb126d11" containerName="storage-initializer"
May 25 14:09:40.207255 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.207226    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a896e5c1-f7e9-4475-95d3-e60edb126d11" containerName="storage-initializer"
May 25 14:09:40.207255 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.207240    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="storage-initializer"
May 25 14:09:40.207255 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.207249    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="storage-initializer"
May 25 14:09:40.207429 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.207305    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="a896e5c1-f7e9-4475-95d3-e60edb126d11" containerName="storage-initializer"
May 25 14:09:40.207429 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.207314    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" containerName="main"
May 25 14:09:40.209228 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.209213    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6knx6/must-gather-8mgsd"
May 25 14:09:40.211604 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.211583    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6knx6\"/\"kube-root-ca.crt\""
May 25 14:09:40.211817 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.211799    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6knx6\"/\"default-dockercfg-tg949\""
May 25 14:09:40.211892 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.211799    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6knx6\"/\"openshift-service-ca.crt\""
May 25 14:09:40.217746 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.217725    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6knx6/must-gather-8mgsd"]
May 25 14:09:40.304592 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.304522    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bcd6f15-1822-4839-8fd2-a511bdf174da-must-gather-output\") pod \"must-gather-8mgsd\" (UID: \"7bcd6f15-1822-4839-8fd2-a511bdf174da\") " pod="openshift-must-gather-6knx6/must-gather-8mgsd"
May 25 14:09:40.304592 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.304584    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnl5m\" (UniqueName: \"kubernetes.io/projected/7bcd6f15-1822-4839-8fd2-a511bdf174da-kube-api-access-bnl5m\") pod \"must-gather-8mgsd\" (UID: \"7bcd6f15-1822-4839-8fd2-a511bdf174da\") " pod="openshift-must-gather-6knx6/must-gather-8mgsd"
May 25 14:09:40.405517 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.405493    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bcd6f15-1822-4839-8fd2-a511bdf174da-must-gather-output\") pod \"must-gather-8mgsd\" (UID: \"7bcd6f15-1822-4839-8fd2-a511bdf174da\") " pod="openshift-must-gather-6knx6/must-gather-8mgsd"
May 25 14:09:40.405636 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.405532    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bnl5m\" (UniqueName: \"kubernetes.io/projected/7bcd6f15-1822-4839-8fd2-a511bdf174da-kube-api-access-bnl5m\") pod \"must-gather-8mgsd\" (UID: \"7bcd6f15-1822-4839-8fd2-a511bdf174da\") " pod="openshift-must-gather-6knx6/must-gather-8mgsd"
May 25 14:09:40.405805 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.405787    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bcd6f15-1822-4839-8fd2-a511bdf174da-must-gather-output\") pod \"must-gather-8mgsd\" (UID: \"7bcd6f15-1822-4839-8fd2-a511bdf174da\") " pod="openshift-must-gather-6knx6/must-gather-8mgsd"
May 25 14:09:40.414643 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.414626    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnl5m\" (UniqueName: \"kubernetes.io/projected/7bcd6f15-1822-4839-8fd2-a511bdf174da-kube-api-access-bnl5m\") pod \"must-gather-8mgsd\" (UID: \"7bcd6f15-1822-4839-8fd2-a511bdf174da\") " pod="openshift-must-gather-6knx6/must-gather-8mgsd"
May 25 14:09:40.518773 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.518753    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6knx6/must-gather-8mgsd"
May 25 14:09:40.642074 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.642034    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6knx6/must-gather-8mgsd"]
May 25 14:09:40.644171 ip-10-0-135-71 kubenswrapper[2573]: W0525 14:09:40.644143    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bcd6f15_1822_4839_8fd2_a511bdf174da.slice/crio-61739edc8cb36e41d24960b232316a373974630a2aeae9813fe966f3626d07f3 WatchSource:0}: Error finding container 61739edc8cb36e41d24960b232316a373974630a2aeae9813fe966f3626d07f3: Status 404 returned error can't find the container with id 61739edc8cb36e41d24960b232316a373974630a2aeae9813fe966f3626d07f3
May 25 14:09:40.888810 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.888752    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6knx6/must-gather-8mgsd" event={"ID":"7bcd6f15-1822-4839-8fd2-a511bdf174da","Type":"ContainerStarted","Data":"61739edc8cb36e41d24960b232316a373974630a2aeae9813fe966f3626d07f3"}
May 25 14:09:40.892848 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.892827    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"f09d7900-a2f5-4867-a5e2-37057d7ce599","Type":"ContainerDied","Data":"c5698089c539be814e576213e4ffd583adcaf374231958e0eb87800380ad2573"}
May 25 14:09:40.892952 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.892871    2573 scope.go:117] "RemoveContainer" containerID="2a3c4c73d8bbf3488c3b0d9d2d5f7b6148f7d3085590d3b83d8ae864e70f9e34"
May 25 14:09:40.892952 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.892910    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 14:09:40.904466 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.904437    2573 scope.go:117] "RemoveContainer" containerID="04f63df08f9711b618b8ed4e68976b24f40d0e569c7cbdfe7acb3b3d184af568"
May 25 14:09:40.919658 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.919638    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 14:09:40.923364 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:40.923342    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 14:09:41.339659 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:41.339615    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f09d7900-a2f5-4867-a5e2-37057d7ce599" path="/var/lib/kubelet/pods/f09d7900-a2f5-4867-a5e2-37057d7ce599/volumes"
May 25 14:09:46.922479 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:46.922406    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6knx6/must-gather-8mgsd" event={"ID":"7bcd6f15-1822-4839-8fd2-a511bdf174da","Type":"ContainerStarted","Data":"4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48"}
May 25 14:09:46.922479 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:46.922443    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6knx6/must-gather-8mgsd" event={"ID":"7bcd6f15-1822-4839-8fd2-a511bdf174da","Type":"ContainerStarted","Data":"fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528"}
May 25 14:09:46.939637 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:09:46.939596    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-6knx6/must-gather-8mgsd" podStartSLOduration=1.056460491 podStartE2EDuration="6.939583628s" podCreationTimestamp="2026-05-25 14:09:40 +0000 UTC" firstStartedPulling="2026-05-25 14:09:40.645817383 +0000 UTC m=+3847.907420205" lastFinishedPulling="2026-05-25 14:09:46.52894053 +0000 UTC m=+3853.790543342" observedRunningTime="2026-05-25 14:09:46.937135996 +0000 UTC m=+3854.198738827" watchObservedRunningTime="2026-05-25 14:09:46.939583628 +0000 UTC m=+3854.201186458"
May 25 14:10:08.499576 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:08.499531    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84964b69b-gj5s2_21ef9931-fc75-4841-b015-3a450c700b77/router/0.log"
May 25 14:10:09.417682 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:09.417651    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84964b69b-gj5s2_21ef9931-fc75-4841-b015-3a450c700b77/router/0.log"
May 25 14:10:10.257138 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:10.257107    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-26l97_1cacaf2f-89c3-4de9-96a6-656d11ffa06a/kuadrant-console-plugin/0.log"
May 25 14:10:10.364150 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:10.364122    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-kkz9n_65f6e7db-0b43-4ce5-a23f-030ad00c428e/limitador/0.log"
May 25 14:10:12.006954 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:12.006917    2573 generic.go:358] "Generic (PLEG): container finished" podID="7bcd6f15-1822-4839-8fd2-a511bdf174da" containerID="fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528" exitCode=0
May 25 14:10:12.007384 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:12.006960    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6knx6/must-gather-8mgsd" event={"ID":"7bcd6f15-1822-4839-8fd2-a511bdf174da","Type":"ContainerDied","Data":"fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528"}
May 25 14:10:12.007384 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:12.007223    2573 scope.go:117] "RemoveContainer" containerID="fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528"
May 25 14:10:12.061424 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:12.061395    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6knx6_must-gather-8mgsd_7bcd6f15-1822-4839-8fd2-a511bdf174da/gather/0.log"
May 25 14:10:15.378521 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:15.378492    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-92l8w_430200e3-29df-46e6-99ce-346a015400ae/global-pull-secret-syncer/0.log"
May 25 14:10:15.466691 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:15.466666    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c65mm_12c6cc0f-fed3-457d-8219-eae48806d598/konnectivity-agent/0.log"
May 25 14:10:15.597131 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:15.597097    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-71.ec2.internal_9a50d4f711fbf255cb7babbc8960a87b/haproxy/0.log"
May 25 14:10:17.558566 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.558506    2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-6knx6/must-gather-8mgsd"]
May 25 14:10:17.559001 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.558756    2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-6knx6/must-gather-8mgsd" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" containerName="copy" containerID="cri-o://4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48" gracePeriod=2
May 25 14:10:17.563528 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.562859    2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-6knx6/must-gather-8mgsd"]
May 25 14:10:17.791391 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.791367    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6knx6_must-gather-8mgsd_7bcd6f15-1822-4839-8fd2-a511bdf174da/copy/0.log"
May 25 14:10:17.791693 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.791677    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6knx6/must-gather-8mgsd"
May 25 14:10:17.793605 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.793579    2573 status_manager.go:895] "Failed to get status for pod" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" pod="openshift-must-gather-6knx6/must-gather-8mgsd" err="pods \"must-gather-8mgsd\" is forbidden: User \"system:node:ip-10-0-135-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6knx6\": no relationship found between node 'ip-10-0-135-71.ec2.internal' and this object"
May 25 14:10:17.818977 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.818921    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnl5m\" (UniqueName: \"kubernetes.io/projected/7bcd6f15-1822-4839-8fd2-a511bdf174da-kube-api-access-bnl5m\") pod \"7bcd6f15-1822-4839-8fd2-a511bdf174da\" (UID: \"7bcd6f15-1822-4839-8fd2-a511bdf174da\") "
May 25 14:10:17.819066 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.818990    2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bcd6f15-1822-4839-8fd2-a511bdf174da-must-gather-output\") pod \"7bcd6f15-1822-4839-8fd2-a511bdf174da\" (UID: \"7bcd6f15-1822-4839-8fd2-a511bdf174da\") "
May 25 14:10:17.821019 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.820990    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcd6f15-1822-4839-8fd2-a511bdf174da-kube-api-access-bnl5m" (OuterVolumeSpecName: "kube-api-access-bnl5m") pod "7bcd6f15-1822-4839-8fd2-a511bdf174da" (UID: "7bcd6f15-1822-4839-8fd2-a511bdf174da"). InnerVolumeSpecName "kube-api-access-bnl5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:10:17.824235 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.824216    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcd6f15-1822-4839-8fd2-a511bdf174da-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "7bcd6f15-1822-4839-8fd2-a511bdf174da" (UID: "7bcd6f15-1822-4839-8fd2-a511bdf174da"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:10:17.920076 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.920046    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnl5m\" (UniqueName: \"kubernetes.io/projected/7bcd6f15-1822-4839-8fd2-a511bdf174da-kube-api-access-bnl5m\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:10:17.920076 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:17.920075    2573 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bcd6f15-1822-4839-8fd2-a511bdf174da-must-gather-output\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 25 14:10:18.028596 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.028576    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6knx6_must-gather-8mgsd_7bcd6f15-1822-4839-8fd2-a511bdf174da/copy/0.log"
May 25 14:10:18.028952 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.028930    2573 generic.go:358] "Generic (PLEG): container finished" podID="7bcd6f15-1822-4839-8fd2-a511bdf174da" containerID="4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48" exitCode=143
May 25 14:10:18.029025 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.028986    2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6knx6/must-gather-8mgsd"
May 25 14:10:18.029025 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.029017    2573 scope.go:117] "RemoveContainer" containerID="4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48"
May 25 14:10:18.031397 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.031372    2573 status_manager.go:895] "Failed to get status for pod" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" pod="openshift-must-gather-6knx6/must-gather-8mgsd" err="pods \"must-gather-8mgsd\" is forbidden: User \"system:node:ip-10-0-135-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6knx6\": no relationship found between node 'ip-10-0-135-71.ec2.internal' and this object"
May 25 14:10:18.037051 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.037033    2573 scope.go:117] "RemoveContainer" containerID="fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528"
May 25 14:10:18.039023 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.038997    2573 status_manager.go:895] "Failed to get status for pod" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" pod="openshift-must-gather-6knx6/must-gather-8mgsd" err="pods \"must-gather-8mgsd\" is forbidden: User \"system:node:ip-10-0-135-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6knx6\": no relationship found between node 'ip-10-0-135-71.ec2.internal' and this object"
May 25 14:10:18.050487 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.050469    2573 scope.go:117] "RemoveContainer" containerID="4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48"
May 25 14:10:18.050809 ip-10-0-135-71 kubenswrapper[2573]: E0525 14:10:18.050788    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48\": container with ID starting with 4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48 not found: ID does not exist" containerID="4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48"
May 25 14:10:18.050885 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.050820    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48"} err="failed to get container status \"4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48\": rpc error: code = NotFound desc = could not find container \"4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48\": container with ID starting with 4742c58c86feb0a2c6dcac73e64a3333cb77adb60d3e80d0ce530c8e60339f48 not found: ID does not exist"
May 25 14:10:18.050885 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.050837    2573 scope.go:117] "RemoveContainer" containerID="fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528"
May 25 14:10:18.051086 ip-10-0-135-71 kubenswrapper[2573]: E0525 14:10:18.051051    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528\": container with ID starting with fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528 not found: ID does not exist" containerID="fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528"
May 25 14:10:18.051127 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:18.051089    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528"} err="failed to get container status \"fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528\": rpc error: code = NotFound desc = could not find container \"fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528\": container with ID starting with fa4d98859591f99a1ade347ae2cce55e5a522a6159cdca52691f51d2c07f0528 not found: ID does not exist"
May 25 14:10:19.337515 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:19.337485    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" path="/var/lib/kubelet/pods/7bcd6f15-1822-4839-8fd2-a511bdf174da/volumes"
May 25 14:10:19.610337 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:19.610313    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-26l97_1cacaf2f-89c3-4de9-96a6-656d11ffa06a/kuadrant-console-plugin/0.log"
May 25 14:10:19.690588 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:19.690535    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-kkz9n_65f6e7db-0b43-4ce5-a23f-030ad00c428e/limitador/0.log"
May 25 14:10:21.005474 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:21.005446    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-j54sp_6e7fa479-726d-460c-b125-97d1adafa418/cluster-monitoring-operator/0.log"
May 25 14:10:21.242207 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:21.242177    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gp65l_01c0855a-d37c-4cd0-bb3a-2713d13f0bc3/node-exporter/0.log"
May 25 14:10:21.267391 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:21.267334    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gp65l_01c0855a-d37c-4cd0-bb3a-2713d13f0bc3/kube-rbac-proxy/0.log"
May 25 14:10:21.290128 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:21.290110    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gp65l_01c0855a-d37c-4cd0-bb3a-2713d13f0bc3/init-textfile/0.log"
May 25 14:10:23.905403 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:23.905369    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 14:10:23.916822 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:23.916798    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/2.log"
May 25 14:10:24.585942 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.585908    2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"]
May 25 14:10:24.586264 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.586247    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" containerName="gather"
May 25 14:10:24.586349 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.586266    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" containerName="gather"
May 25 14:10:24.586349 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.586292    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" containerName="copy"
May 25 14:10:24.586349 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.586301    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" containerName="copy"
May 25 14:10:24.586504 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.586376    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" containerName="gather"
May 25 14:10:24.586504 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.586388    2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="7bcd6f15-1822-4839-8fd2-a511bdf174da" containerName="copy"
May 25 14:10:24.588453 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.588429    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.590656 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.590634    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tl2jt\"/\"kube-root-ca.crt\""
May 25 14:10:24.590785 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.590762    2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tl2jt\"/\"default-dockercfg-kltt4\""
May 25 14:10:24.590846 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.590804    2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tl2jt\"/\"openshift-service-ca.crt\""
May 25 14:10:24.596623 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.596601    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"]
May 25 14:10:24.672018 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.671997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz5cl\" (UniqueName: \"kubernetes.io/projected/2b11d7d0-84f4-4a2c-94c4-a911e2624913-kube-api-access-tz5cl\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.672128 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.672033    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-proc\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.672128 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.672052    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-lib-modules\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.672206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.672123    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-sys\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.672206 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.672163    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-podres\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.772628 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.772600    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tz5cl\" (UniqueName: \"kubernetes.io/projected/2b11d7d0-84f4-4a2c-94c4-a911e2624913-kube-api-access-tz5cl\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.772731 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.772640    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-proc\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.772731 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.772657    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-lib-modules\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.772731 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.772688    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-sys\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.772731 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.772720    2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-podres\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.772893 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.772741    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-proc\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.772893 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.772788    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-sys\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.772893 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.772833    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-lib-modules\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.772893 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.772855    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2b11d7d0-84f4-4a2c-94c4-a911e2624913-podres\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.780525 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.780500    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz5cl\" (UniqueName: \"kubernetes.io/projected/2b11d7d0-84f4-4a2c-94c4-a911e2624913-kube-api-access-tz5cl\") pod \"perf-node-gather-daemonset-7bhn4\" (UID: \"2b11d7d0-84f4-4a2c-94c4-a911e2624913\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.898822 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.898765    2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:24.902963 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:24.902942    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-ffjg6_05c02256-ea7d-4a74-8f08-a16384769838/volume-data-source-validator/0.log"
May 25 14:10:25.023625 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:25.023601    2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"]
May 25 14:10:25.026017 ip-10-0-135-71 kubenswrapper[2573]: W0525 14:10:25.025992    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2b11d7d0_84f4_4a2c_94c4_a911e2624913.slice/crio-aae1118d8fba8e8db93f0b2d8c1d34f2880ee9231996a7ef1f546a571778c718 WatchSource:0}: Error finding container aae1118d8fba8e8db93f0b2d8c1d34f2880ee9231996a7ef1f546a571778c718: Status 404 returned error can't find the container with id aae1118d8fba8e8db93f0b2d8c1d34f2880ee9231996a7ef1f546a571778c718
May 25 14:10:25.054195 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:25.054169    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4" event={"ID":"2b11d7d0-84f4-4a2c-94c4-a911e2624913","Type":"ContainerStarted","Data":"aae1118d8fba8e8db93f0b2d8c1d34f2880ee9231996a7ef1f546a571778c718"}
May 25 14:10:25.749205 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:25.749174    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q6pgt_e0e32442-e801-4fcb-8c48-74b788e18965/dns/0.log"
May 25 14:10:25.772503 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:25.772482    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q6pgt_e0e32442-e801-4fcb-8c48-74b788e18965/kube-rbac-proxy/0.log"
May 25 14:10:25.849933 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:25.849915    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wfc8p_ab2520dc-3572-44d2-b557-7d15a0bc0c45/dns-node-resolver/0.log"
May 25 14:10:26.058928 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:26.058866    2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4" event={"ID":"2b11d7d0-84f4-4a2c-94c4-a911e2624913","Type":"ContainerStarted","Data":"7dbca553491a9bc761c1558237d92349a0a6cb4ec7d24cd88aaaa33ddf1d63a5"}
May 25 14:10:26.059223 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:26.059006    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:26.075154 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:26.075097    2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4" podStartSLOduration=2.07508343 podStartE2EDuration="2.07508343s" podCreationTimestamp="2026-05-25 14:10:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:10:26.074570904 +0000 UTC m=+3893.336173732" watchObservedRunningTime="2026-05-25 14:10:26.07508343 +0000 UTC m=+3893.336686263"
May 25 14:10:26.406843 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:26.406776    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f78nn_880fa08b-8cfb-45ce-8c9c-b63dd8896b39/node-ca/0.log"
May 25 14:10:27.381083 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:27.381053    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84964b69b-gj5s2_21ef9931-fc75-4841-b015-3a450c700b77/router/0.log"
May 25 14:10:27.864316 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:27.864288    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjzcb_2f58bc38-d685-405e-aa7e-7de7a86001ee/serve-healthcheck-canary/0.log"
May 25 14:10:28.353521 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:28.353481    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-sqk9d_a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59/insights-operator/1.log"
May 25 14:10:28.355851 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:28.355824    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-sqk9d_a7a4f5bc-df6e-4bea-9f95-8ec41af6fb59/insights-operator/0.log"
May 25 14:10:28.468096 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:28.468074    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hlx8q_c2367bd3-2167-4ca1-be1c-01c25e1bb1d7/kube-rbac-proxy/0.log"
May 25 14:10:28.490978 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:28.490945    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hlx8q_c2367bd3-2167-4ca1-be1c-01c25e1bb1d7/exporter/0.log"
May 25 14:10:28.516559 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:28.516532    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hlx8q_c2367bd3-2167-4ca1-be1c-01c25e1bb1d7/extractor/0.log"
May 25 14:10:31.194620 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:31.194592    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f6cfd7d7-p4xl7_b83b4cb1-3e33-459b-b6f9-04e4e7315c6d/manager/0.log"
May 25 14:10:31.245164 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:31.245137    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-bfc7f696d-8xw5x_226140ab-8d9e-4b33-a30e-45779b3dbb1a/openshift-lws-operator/0.log"
May 25 14:10:32.071905 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:32.071873    2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-7bhn4"
May 25 14:10:33.535584 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:33.535557    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 14:10:33.547163 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:33.547142    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jwb2k_ca0dc551-06fa-414c-a3e3-28f8d0b2e382/console-operator/1.log"
May 25 14:10:37.298031 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:37.297989    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-8kgql_a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8/kube-storage-version-migrator-operator/1.log"
May 25 14:10:37.299710 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:37.299683    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-8kgql_a3bfc893-a8bd-4f6b-ab4b-adcdd82c96f8/kube-storage-version-migrator-operator/0.log"
May 25 14:10:38.761112 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:38.761085    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mkkfs_cda7f987-514d-47c2-b386-52b0eba3d858/kube-multus-additional-cni-plugins/0.log"
May 25 14:10:38.783610 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:38.783586    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mkkfs_cda7f987-514d-47c2-b386-52b0eba3d858/egress-router-binary-copy/0.log"
May 25 14:10:38.804693 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:38.804677    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mkkfs_cda7f987-514d-47c2-b386-52b0eba3d858/cni-plugins/0.log"
May 25 14:10:38.825534 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:38.825515    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mkkfs_cda7f987-514d-47c2-b386-52b0eba3d858/bond-cni-plugin/0.log"
May 25 14:10:38.846832 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:38.846813    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mkkfs_cda7f987-514d-47c2-b386-52b0eba3d858/routeoverride-cni/0.log"
May 25 14:10:38.871344 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:38.871324    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mkkfs_cda7f987-514d-47c2-b386-52b0eba3d858/whereabouts-cni-bincopy/0.log"
May 25 14:10:38.892734 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:38.892711    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mkkfs_cda7f987-514d-47c2-b386-52b0eba3d858/whereabouts-cni/0.log"
May 25 14:10:38.934317 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:38.934292    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rckll_d5eaa3b3-e7d0-466d-8e08-0e887a745119/kube-multus/0.log"
May 25 14:10:39.068196 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:39.068141    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tkljt_c1774200-9350-464d-a3b8-2be8812bcde4/network-metrics-daemon/0.log"
May 25 14:10:39.090912 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:39.090887    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tkljt_c1774200-9350-464d-a3b8-2be8812bcde4/kube-rbac-proxy/0.log"
May 25 14:10:39.978850 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:39.978816    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-684s7_0f3d6ddb-a174-491b-b713-b4552899da35/ovn-controller/0.log"
May 25 14:10:40.027971 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:40.027947    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-684s7_0f3d6ddb-a174-491b-b713-b4552899da35/ovn-acl-logging/0.log"
May 25 14:10:40.049959 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:40.049938    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-684s7_0f3d6ddb-a174-491b-b713-b4552899da35/kube-rbac-proxy-node/0.log"
May 25 14:10:40.076915 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:40.076893    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-684s7_0f3d6ddb-a174-491b-b713-b4552899da35/kube-rbac-proxy-ovn-metrics/0.log"
May 25 14:10:40.100385 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:40.100368    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-684s7_0f3d6ddb-a174-491b-b713-b4552899da35/northd/0.log"
May 25 14:10:40.121378 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:40.121362    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-684s7_0f3d6ddb-a174-491b-b713-b4552899da35/nbdb/0.log"
May 25 14:10:40.142484 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:40.142466    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-684s7_0f3d6ddb-a174-491b-b713-b4552899da35/sbdb/0.log"
May 25 14:10:40.323038 ip-10-0-135-71 kubenswrapper[2573]: I0525 14:10:40.322993    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-684s7_0f3d6ddb-a174-491b-b713-b4552899da35/ovnkube-controller/0.log"